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ABSTRACT 


This  thesis  details  the  engineering,  design  and 
implementation  of  a  real-time,  distributed,  application 
emulator  system  (AE  system)  .  The  project  had  two  main  goals 
for  the  tool:  emulation  of  real-time  distributed  systems,  and 
as  a  programmable  resource  consumer.  The  AE  system  is 
currently  being  used  in  the  HiPer-D  test  bed  to  activate  a 
resource  leveling  tool  that  monitors  several  software 
components  for  real-time  response.  The  AE  system  is  highly 
flexible  and  can  be  used  in  the  context  of  a  variety  of 
network  topologies  and  system  loading  options.  The  results 
presented  show  that  the  AE  system  also  emulates  distributed 
systems . 
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I. 


INTRODUCTION 


This  thesis  describes  the  design,  implementation,  and 
evaluation  of  a  software  tool  that  is  capable  of  emulating 
real-time  distributed  applications.  The  tool  is  formally- 
known  as  the  Application  Emulator  (AE)  system,  and  the 
primary  goal  of  the  project  is  to  emulate  Real-Time 
Distributed  Systems  (RTDS) .  This  is  achieved  by  providing 
software  that  can  be  easily  configured  to  resemble  a 
particular  application,  chosen  from  a  wide  range  of  real¬ 
time  applications.  The  AE  system  is  not  meant  to  provide 
the  functionality  of  real-time  applications,  but  rather  to 
imitate  the  resource  usage  patterns  of  such  applications. 

The  AE  was  developed  using  an  iterative  process.  Some 
iterations  added  functionality  to  the  AE  and  allowed  the  AE 
system  to  emulate  a  wider  range  of  RTDS .  Other  iterations 
concentrated  on  generalizing  the  design,  emphasizing  the 
concept  of  software  reuse.  These  iterations  tended  to 
simplify  the  design.  The  final  design  has  a  scalable 
architecture  that  can  be  configured  to  emulate  RTDS 
containing  many  components,  each  perhaps  executing  on  a 
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different  system,  and  each  perhaps  having  real-time 
deadlines . 

A.  MOTIVATION 

The  research  performed  for  this  thesis  contributes 
substantially  to  the  Naval  Surface  Warfare  Center's  (NSWC) 
High  Performance  Distributed  (HiPer-D)  computing  project. 
The  HiPer-D  project  is  currently  developing  a  prototype, 
next -generation  high  performance  Anti  Air  Warfare  Weapon 
Control  System  (WCS) .  The  project  is  focused  on  determining 
whether  Commercial  Off  The  Shelf  (COTS)  systems  can  meet  the 
real-time,  scalability  and  fault  tolerance  requirements  of 
such  applications.  If  successful,  the  move  to  COTS  will 
offer  several  advantages  over  dedicated  systems  including: 

•  lower  software  and  hardware  costs, 

•  higher  performance  (faster  computationally) 
computers  in  terms  of  processing  power, 

•  reduced  hardware  upgrade  times,  and 

•  user  familiarity  with  interfaces  and  components. 

The  move  to  COTS  to  support  these  applications 
represents  a  major  paradigm  shift.  The  currently  fielded 
set  of  WCS  applications  is  supported  by  special-purpose, 
dedicated  hardware  (i.e.,  computers).  The  communication 
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between  components  is  supported  by  point-to-point  dedicated 
connections.  The  set  of  fielded  applications  comprises  a 
large  and  complex  entity.  The  prototype,  currently  being 
developed  and  analyzed  in  the  HiPer-D  laboratory  at  NSWC, 
while  also  large,  does  not  encompass  the  entire 
functionality  of  the  fielded  application.  Adding  the  full 
functionality  to  the  existing  laboratory  code  would  require 
a  substantial  investment.  Therefore,  an  AE  system  that  can 
be  configured  to  accurately  emulate  the  software  components 
that  are  part  of  the  fielded  system  but  are  not  part  of  the 
prototype  would  aid  in  analyzing  the  suitability  of  COTS  for 
these  applications  at  a  fraction  of  the  cost.  An  AE  used  in 
this  context  must  accurately  mimic  the  loads  that  would  be 
placed  on  the  computing  and  communication  resources  by  the 
missing  components. 

Part  of  the  HiPer-D  mission  is  to  determine  whether  the 
next  generation  WCS  can  meet  its  requirements  if  implemented 
using  COTS  components.  Furthermore,  if  the  current  COTS 
systems  do  not  meet  the  needs  of  such  applications,  the 
HiPer-D  project  must  identify  the  areas  of  today's 
technology  that  fail  to  provide  such  support.  Finally,  when 
such  areas  are  identified,  the  project  may  also  suggest 
avenues  for  new  COTS  technologies  that  will  better  meet 
Navy's  application  requirements. 
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This  motivation  helps  explain  the  basic  requirements  of 
the  AE  project.  An  application  observed  from  an  external 
vantage  point  has  a  CPU  usage  pattern  (or  profile) ,  a 
network  usage  pattern,  and  a  memory  usage  pattern,  in 
addition  to  usage  patterns  for  some  less  obvious  resources 
such  as  file  server  access.  The  main  goal  of  this  research, 
therefore,  is  to  design  and  implement  software  that  can  be 
easily  configured  to  accurately  imitate  the  resource  usage 
patterns  of  WCS  software.  Obviously,  the  algorithms  that 
will  be  used  by  next -generation  WCS  may  be  different  from 
those  used  in  today's  systems.  Therefore,  the  AE  must  be 
able  to  imitate  not  only  the  usage  pattern  of  today's  WCS 
applications,  but  tomorrow's  as  well. 

The  algorithms  used  in  WCS  applications  will  likely 
change  over  time,  yet  based  upon  existing  WCS  applications 
[T3]  ,  it  is  clear  that  any  algorithm  that  performs  weapon 
control  will  fall  into  a  class  of  applications  known  as 
periodic,  real-time  applications.  The  main  characteristics 
of  this  class  of  applications  are  that  they  repeatedly 
receive  sensor  or  pre-processed  information,  execute  one  or 
several  f iltering-type  algorithms,  and  report  an  answer 
before  a  deadline.  Such  applications  are  both  CPU  and 
network - intens ive .  Therefore,  the  design  for  this  AE 
project  has  focused  on  two  main  areas:  the  ability  to 
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replicate  CPU  usage  patterns  and  the  ability  to  replicate 
network  usage  patterns.  Additionally,  the  temporal 

relationship  between  these  two  uses  must  also  be  replicated. 
As  part  of  this  thesis,  we  discuss  how  this  design  may  be 
expanded  in  the  future  to  include  replicating  usage  patterns 
for  other  resources. 

Using  the  AE  system,  together  with  the  components  of 
the  next-generation  WCS  that  have  already  been  implemented, 
will  provide  a  higher  level  of  confidence  concerning 
conclusions  about  the  ability  of  COTS  to  support  the  next- 
generation  WCS.  Without  the  AE  system,  or  alternatively  the 
costly  implementation  of  the  rest  of  the  functionality  of 
the  application,  the  adequacy,  strengths,  and  weaknesses  of 
the  COTS  system  might  be  unknown.  For  these  reasons,  the  AE 
is  an  important  tool  for  the  HiPer-D  project. 

B.  AE  SYSTEM  REQUIREMENTS 

The  AE  system  was  designed  and  built  to  emulate  RTDS . 
As  such,  it  has  the  following  high-level  requirements: 

•  Its  architecture  must  be  both  distributed  and 
scaleable . 

•  It  must  be  written  in  a  language  that  is  portable, 
supports  multiple  threads.  It  must  be  designed  to 
reduce  life  cycle  costs. 
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•  It  must  be  capable  of  being  configured  to  emulate 
real-time  applications  that  have  periodic  deadlines. 

•  It  must  be  possible  to  determine  whether  performance 
requirements,  such  as  deadlines,  are  met. 

•  It  must  produce  similar  resource  loads  when  run 
repeatedly  with  the  same  parameters.  In  particular, 
it  must  produce  repeatable  CPU  and  network  usage 
patterns . 

Compliance  with  the  above  requirements  is  discussed  below. 

The  AE  system  must  be  able  to  mimic  applications 
comprised  of  many  components,  although  the  separate 
components  may  execute  on  different  systems.  In  other 
words,  the  AE  must  consist  of  components  that  can  be 
replicated,  individually  configured,  and  distributed.  The 
current  requirement  is  that  the  AE  must  be  able  to  mimic  the 
operation  of  an  application  consisting  of  at  least  twenty 
different  communicating  components,  which  may  be  running  on 
any  number  of  systems  that  support  the  defacto  LAN  standard, 
TCP/IP. 

The  second  requirement  is  that  the  emulator  must 
include  a  wide  range  of  features.  Although  many  other 
modern  languages  meet  the  language  requirements  of  the  AE 
project,  the  decision  to  develop  the  AE  in  Ada95  was  largely 
due  to  a  previous  Navy  requirement  that  stipulated  the 
programming  language  that  must  be  used  for  Naval  real-time 
applications . 
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The  third  requirement  deals  with  the  real-time 
characteristics  of  the  AE  project.  For  the  purpose  of  this 
thesis,  we  will  use  the  two  terms  hard  real-time  and  soft 
real-time  as  they  are  commonly  used  in  the  literature.  A 
component  with  hard  real-time  deadlines  must  complete  its 
periodic  work  before  the  deadline  for  each  period  in  order 
to  satisfy  system  requirements.  Soft  real-time  applications 
meet  their  requirements  if  the  statistical  mean  of  the 
sample  distribution  of  response  times  satisfies  the  deadline 
constraints  [Lui73]  .  For  the  purpose  of  this  thesis's  AE, 
hard  real-time  constraints  were  interpreted  to  mean  that 
missed  deadlines  must  be  reported.  In  many  applications 
with  real-time  periodic  deadlines,  the  deadline  of  the 
previous  period  is  also  the  start  of  the  next  time  period 
[Hart 89]  . 

In  order  to  meet  the  last  requirement  described  above, 
the  AE  must  have  a  repeatable  way  to  apply  a  load  to  the 
network  and  the  CPU,  as  well  as  to  other  resources.  The 
networking  load  requirement  requires  that  the  AE  must  be 
configurable  to  allow  the  components  to  send  and  receive 
messages  to  one  another  in  such  a  way  that  the  dynamic 
message-passing  topology  can  easily  be  specified.  The  CPU 
loading  specification  should  ideally  be  independent  of  the 
speed  and  instruction  set  of  the  processor. 
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c. 


ORGANIZATION 


The  remainder  of  this  thesis  is  organized  as  follows: 
Chapter  II  contains  an  overview  of  related  work.  Chapter 
III  discusses  the  details  of  the  AE  and  its  components.  It 
covers  the  desired  features  of  the  AE,  first  at  a  system 
level,  and  second  it  includes  a  detailed  discussion  of  the 
components  in  an  AE  unit.  Chapter  IV  provides  an  analysis 
of  the  AE  system  including  an  overview  of  the  emulation 
steps.  The  chapter  includes  a  section  on  the  application 
being  emulated  (EADSIM) .  The  chapter  finishes  with  a 
section  on  results  optioned  and  an  analysis  of  the  AE 
system.  Chapter  V  concludes  the  thesis  by  discussing  some 
lessons  learned  while  developing  the  AE,  by  suggesting 
future  work  for  the  AE,  and,  by  contrasting  the  AE  with 
related  tools  for  real-time  system  emulation. 
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II .  RELATED  WORK 


This  chapter  describes  several  simulation  and  emulation 
tools  that  are  closely  related  to  the  AE  system.  A 
comparison  between  the  tools  discussed  here  and  the  AE  is 
presented  in  Chapter  V  after  the  reader  has  had  an 
opportunity  to  read  Chapters  III  and  IV  and  has  a  clearer 
understanding  of  the  AE  system. 

A.  DYNBENCH 

DynBench  [WELC98]  is  a  benchmark  suite  that  was 
designed  to  emulate  a  portion  of  the  prototype  next 
generation  Anti-Air  Warfare,  or,  as  it  is  better  known, 
HiPer-D  [T3]  .  Therefore,  DynBench  is  a  real-time 
distributed  system  with  Quality  of  Service  (QoS) 
requirements.  Just  like  HiPer-D,  DynBench  is  a  system  that 
allows  certain  software  components  of  its  distributed  system 
to  be  relocated  during  operation.  Relocating  a  software 
component  means  moving  it  from  one  computer  to  another 
computer.  Relocation  of  runtime  components  is  a  feature 
that  requires  an  outside  action  (i.e.,  some  other  software 
component  to  act  on  resource  data,  and  to  actually  kill  and 
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restart  the  DynBench  components  being  moved)  and  is  normally 
in  response  to  problems  related  to  loading  or  faults1.  For 
example,  any  or  all  the  components  that  make  up  a  DynBench 
run-time  system  can  be  relocated  while  DynBench  is 
operating. 

DynBench  maintains  Quality  of  Service  (QoS)  data  and 
provides  an  Application  Programming  Interface  (API)  to  allow 
a  Resource  Management  System  (RMS)  or  some  other  process 
access  to  the  QoS  data.  This  data  can  be  used  for 
intelligent  run-time  adaptation. 

The  primary  load  on  a  DynBench  system  (and  the  tactical 
systems  that  it  emulates)  is  a  function  of  the  number  of 
objects,  or  radar  tracks  in  the  radar  view.  Radar  tracks 
have  various  attributes  such  as  speed,  heading,  and 
identification,  i.e.,  friend  or  foe.  DynBench  provides  two 
methods  for  changing  the  track  load.  The  first  is  based  on 
time,  where  the  number  of  tracks  in  the  system  is  increased 
or  decreased  as  a  function  of  time.  The  second  allows  the 
number  of  tracks  to  be  instantaneously  changed  by  an  issuing 
interactive  command. 


1  A  fault  can  be  a  computer  system  that  fails  or  a  network 
segment  that  also  fails  and  causes  a  computer  system  to  be 
isolated. 
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Other  characteristics  of  radar  tracks  that  DynBench 
must  account  for  include  initialization  (where  they  start) 
and  heading  (where  they  are  going) .  These  characteristics 
are  'important  when  studying  a  tactical  system,  but  are  much 
less  important  if  one  is  using  DynBench  as  a  loading  tool 
and  not  as  a  tactical  system. 

The  DynBench  benchmark  suite  uses  simulated  data  in  its 
messages  and  processes  this  data  with  accepted  algorithms. 
In  contrast,  most  real-time  benchmarks  use  a  synthetic 
workload2  created  by  calling  existing  CPU  loading  benchmarks 
(e.g..  Whetstone)  for  workloads.  This  places  DynBench 
somewhere  between  a  general-purpose  benchmark  and  an  actual 
system. 

B .  CARFF ' S  EMULATOR 

Paul  Carff  [CARF99]  performed  research  aimed  at 
determining  how  much  data  are  needed  about  an  application' s 
run  time  resource  usage  in  order  to  predict  how  it  will 
perform  on  different  platforms  (i.e.,  different  processor, 
memory  and  operating  system  configurations) .  Collecting  the 

2  A  synthetic  workload  is  a  nonreal-world  program  that 
usually  exercises  one  aspect  of  a  system.  An  example  is  the 
Whetstone  benchmark  that  performs  operations  on  floating 
point  numbers . 
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right  amount  of  resource  information  is  a  difficult  problem. 
If  not  enough  information  is  obtained,  then  the  resulting, 
prediction  may  be  incorrect  by  more  than  an  acceptable 
percentage.  Large  amounts  of  data  can  put  a  strain  on  the 
run-time  performance  of  the  system  under  study  and  increase 
the  problem  of  managing  and  processing  the  data.  The 
optimum  level  of  data  collection  should  allow  prediction  to 
occur  within  a  certain  level  of  accuracy  (e.g.,  plus  or 
minus  10%)  at  some  level  of  consistency  (e.g.,  90%  of  the 

time)  . 

The  application  Carff  utilized  was  a  distributed 
message  passing  application  that  allows  for  a  configurable 
number  of  components.  Systems  such  as  MSHN  (section  II. E) 
are  designed  to  use  runtime  data  similar  to  that  collected 
by  Carff  to  predict  resource  requirements,  and,  when 
possible,  completion  time  estimates  for  applications.  MSHN 
was  intended  to  use  this  information  to  decide  where  an 
application  should  run,  while  accounting  for  many  factors 
including : 

•  the  deadline  of  an  application, 

•  the  current  state  of  the  system,  and 

•  other  pending  jobs. 


12 


Carff  developed  an  emulator  to  validate  his  thesis. 
His  emulator,  developed  in  Java,  contains  four  modules,  each 
of  which  is  always  executed  within  its  own  thread.  A  brief 
description  of  each  module  is  given  below. 

•  Main.  This  module  receives  the  information  needed 
to  configure  the  other  three  modules.  It  also  must 
wait  until  the  other  modules  complete  before 
exiting. 

•  Calculation.  This  module  performs  the  CPU  loading 
by  multiplying  two  100  x  100  matrices.  It  instructs 
the  sending  modules  to  send  messages  based  on  either 
a  fixed  interval  or  a  statistical  distribution  on 
the  progress  made  in  finishing  the  multiplication. 

•  Sending.  Each  instance  of  Carff 's  emulator  has  a 

send  thread  for  every  other  emulator  in  the  test . 
If  there  are  n  (e.g.,  n=5)  copies  of  the  emulator 

running  then  each  will  have  (n-1)  copies  of  the  send 
module  (e.g.,  4)  .  Each  module  sends  messages  to 

only  one  remote  emulator. 

•  Receive.  This  module  is  very  similar  to  the  Send 
module.  Each  emulator  will  have  n-1  copies  of  the 
receive  modules  running.  Each  receive  module  is 
dedicated  to  receiving  from  a  single  remote  sender. 


The  emulator  runs  until  all  threads  have  completed 
their  work.  This  work  includes  reception  of  all  messages 
and,  of  course,  finishing  work  that  emulates  CPU  loading. 
Only  when  the  calculation,  and  receive  and  send  threads  have 
completed  will  the  main  module  terminate. 
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c. 


PETRI  NETS 


Petri  nets  can  be  used  as  a  tool  for  the  indirect  study 
of  a  system  [PETE81]  .  The  first  step  to  utilizing  Petri 
nets  is  to  create  a  model  of  an  existing  system  by 
incorporating  all  important  features  of  that  system  into  the 
model.  Then,  the  model,  which  is  mathematical  in  nature, 
can  be  analyzed  to  learn  more  about  the  actual  system. 

Many  different  systems  can  benefit  from  this  indirect 
method  of  study.  There  are  many  cases  when  the  Petri  net 
method  can  work  better  than  studying  the  actual  system.  For 
example,  astronomy  (where  the  times  involved  in  the  actual 
system  are  too  great)  and  sociology  (where  studies  might 
cause  ethical  problems)  are  eminently  suitable  for  studies 
via  Petri  nets  [PETE81] .  The  DynBench  benchmark  suite  is  an 
example  of  a  model  representing  a  system  that  is  not  readily 
available  for  experimentation  since  the  software  is 
considered  proprietary  and  sensitive  (for  national  security) 
in  nature.  The  operating  signature  (resource  utilization  of 
the  COTS  components)  does  not  have  the  same  level  of 
classification  and  can  be  used  in  the  model. 
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Well-constructed  models  are  necessary  for  useful  Petri 
nets.  If  a  model  is  poorly  constructed  then  the  resulting 
conclusions  can  and  probably  will  be  incorrect.  If  a  model 
is  well  constructed  and  correctly  represents  the  major 
features  of  the  real  world  system,  then  the  Petri  net  study 
can  yield  usable  results.  The  interaction  between  the 
components  (which  may  be  large  complex  systems  themselves) 
must  be  preserved  in  the  model.  Each  component  of  the 
system  has  a  separate  behavior  as  well  as  an  interaction 
with  the  other  components .  That  behavior  may  change  over 
time  and/or  events  (the  current  state)  . 

D .  HARTS  TONE  BENCHMARK 

Hartstone  is  a  real-time  benchmark  suite.  It  was 
initially  proposed  as  a  specification  [HART89]  for  a 
benchmark  suite,  but  later  was  developed  into  a  working 
tool.  The  premise  of  the  Hartstone  benchmark  is  that 
developers  can  prototype  a  proposed  real-time  system,  and 
then  execute  that  prototype  on  the  intended  computer 
hardware.  This  allows  the  designers  to  quickly  see  how  the 
actual  system  will  perform  within  some  margin  of  error  when 
fielded.  Before  Hartstone,  developers  would  either  wait 
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until  the  system  was  built  or  would  have  developed  a 
prototype  themselves  in  order  to  test  system  response. 

The  Hartstone  Benchmark  provides  a  series  of  tests  to 
conduct  on  a  real-time  system  [HART89]  .  The  five  tests 
defined  by  Hartstone  [HART89]  are  given  below: 

•  PH  Series:  Periodic  Tasks,  Harmonic  Frequencies 

•  PN  Series:  Periodic  Tasks,  Non-Harmonic  Frequencies 

•  AH  Series:  PH  Series  with  Aperiodic  Processing  Added 

•  SH  Series:  PH  Series  with  Synchronization 

•  SA  Series:  PH  Series  with  Aperiodic  Processing  and 
Synchronization 

Harmonic  frequencies  means  that  all  the  periodic  tasks 
have  a  common  base  frequency.  The  task  with  the  shortest 
time  period,  operates  at  the  base  frequency.  All  the  other 
task's  periods  are  some  multiple  of  the  base  frequency.  For 
example,  if  a  system  includes  three  tasks,  and  they  have  the 
following  periods:  Task  One  is  one  second,  Task  Two  is  two 
seconds  and  Task  Three  is  four  seconds,  then  these  tasks  are 
harmonic  with  a  base  frequency  of  one  second  (i.e..  Task  One 
is  the  task  that  runs  at  the  base  frequency)  .  When  the 
tasks  are  synchronized  and  harmonic  they  all  start  at  the 
same  time,  so  every  fourth  second  all  three  tasks  will  start 
an  execution  cycle. 
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A  test  utilizing  the  Hartstone  benchmark  continues 
until  the  system  misses  a  hard  real-time  deadline.  If, 
during  the  first  run  of  the  system  all  hard  deadlines  are 
met,  then  one  part  of  the  system  is  changed  to  make  meeting 
the  deadlines  during  the  next  run  more  difficult.  This 
continues  until  the  system  fails  to  meet  a  hard  deadline. 
The  Hartstone  benchmark  is  intended  to  measure  the  breakdown 
point  of  a  real-time  system  [HART92] .  Hartstone  benchmark 
results  allow  a  real-time  system  designer  to  know  before 
software  development  if  the  end  product  could  operate  as 
specified  in  terms  of  real-time  response. 


E.  MSHN 


The  Management  System  for  Heterogeneous  Networks  (MSHN) 
was  a  research  project  to  develop  a  Resource  Management 
System  (RMS)  .  One  goal  of  a  RMS  system  is  to  make  a  set  of 
distributed  computational  resources  (heterogeneous  in  MSHN's 
case)  look  and  act  like  one  virtual  machine  [HENS99]  . 
Distributed  Operating  Systems  are  also  tools  that  attempt  to 
make  a  set  of  networked  machines  look  like  one  virtual 
system.  One  major  difference  between  a  RMS  and  a 
distributed  operating  system  is  that  the  RMS  does  not  manage 
system  resources.  That  task  belongs  to  each  local  operating 
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system.  Thus,  an  RMS  provides  a  mechanism  for  intelligently 
assigning  applications  to  a  computing  system  selected  from 
set  of  computing  resources . 

One  objective  of  MSHN  was  to  attempt  to  meet  QoS 
requirements  by  supporting  adaptive  applications  [HENS99] . 
Adaptive  applications  allow  for  different  levels  of  fidelity 
in  the  output.  For  example,  directions  to  a  local  airport 
can  be  delivered  as  a  color  map,  a  black  and  white  map,  or, 
in  the  simplest  case  as  an  ASCII  description.  If  the 
completion  time  is  critical,  then  meeting  that  deadline  (in 
this  case,  perhaps  catching  a  plane)  is  the  primary  concern 
and  the  quality  (in  terms  of  display)  of  the  result  is 
secondary.  The  intended  MSHN  goal  was  to  meet  deadlines 
with  the  best  quality  results,  but  if  available  resource 
levels  would  not  permit  the  deadline  to  be  met,  then  to 
adapt  one  or  more  of  the  applications  so  that  the  deadline 
could  be  realized. 

The  adaptable  features  described  for  MSHN  appear 
promising  for  future  applications.  The  idea  of  applications 
that  dynamically  adapt  themselves  through  tools  like  MSHN, 
allowing  several  competing  applications  to  all  meet  their 
deadlines,  would  be  a  major  advancement  over  today's  static 
software  systems. 
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III.  APPLICATION  EMULATOR  SYSTEM  AND  COMPONENTS 


Chapter  III  has  the  following  organization.  The 
introduction  contains  a  high  level  overview  of  the  AE 
system.  Section  B,  contains  background  information  and  the 
constraints  on  the  AE  project.  The  chapter  finishes  with  a 
section  giving  the  functional  block  diagram  of  an  AE  unit. 
This  section  also  contains  a  detailed  description  of  the 
components  that  make  up  an  AE  unit. 


A.  INTRODUCTION 


In  developing  the  AE,  the  primary  goal  was  to  keep  the 
components  of  the  AE  unit  fairly  generic  and  flexible. 
Although  this  may  seem  to  constrain  the  capability  of  the  AE 
system,  it  will  be  shown  that  the  AE  system  can  emulate  a 
large  Real  Time  Distributed  System  (RTDS) .  Large  system 
emulation  is  possible  because  each  AE  unit  can  mimic  most 
real-time  components  and  large  systems  can  be  constructed  by 
interconnecting  them  in  a  wide  range  of  topologies. 

In  the  HiPer-D  program  one  of  the  early  needs  for  an 
emulation  or  simulation  tool  was  to  place  additional  load  on 
partial  implementations  of  large  real-time  systems.  In  this 
role,  the  AE  system  would  emulate  missing  or  unfinished 
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components  that  are  part  of  a  larger  system.  Its  role  would 
be  to  emulate  the  missing  component's  resource  usage  profile 
and  thereby  create  the  illusion  of  having  the  component 
present.  The  resulting  system,  consisting  of  actual  code 
and  the  AE  system,  can  then  be  studied.  The  ease  of 
modifying  the  AE  system  usage  profile  allows  a  wide  range  of 
tests  to  be  conducted  much  like  a  Monte  Carlo  simulation. 
In  this  role,  the  AE  system  is  utilized  as  a  tool  for 
quickly  prototyping  real-time  components. 

The  CPU  cycles  consumed  by  an  AE  unit  can  be  divided 
into  two  areas,  overhead  and  emulation.  The  overhead  CPU 
usage  is  introduced  by  the  AE  as  it  executes  an  emulation 
and  is  a  necessary  aspect  of  any  emulation  package.  The  AE 
system,  at  a  system  and  component  level  was  carefully 
designed  and  coded  to  minimize  CPU  processing  while 
maintaining  a  acceptable  level  of  functionality.  The  CPU 
emulation,  which  is  central  to  application  emulation,  is 
accomplished  by  placing  a  synthetic  workload  on  a  system. 

Each  AE  unit  provides  the  ability  to  emulate  the 
resource  utilization  of  the  following  system  resources:  CPU, 
network  and  memory.  The  rest  of  this  chapter  will  cover  the 
detailed  requirements  and  the  design  of  the  Application 
Emulator.  Further,  it  will  show  how  the  design  and 
implementation  meet  the  requirements. 
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B .  PROJECT  REQUIREMENTS 


The  initial  goal  of  the  AE  project  was  to  provide  an 
application  emulator  that  could  mimic  the  resource 
utilization  of  existing  or  planned  real-time  applications. 
A  working  group  from  the  HiPer-D  project  formulated  the 
following  high-level  requirements  for  the  AE  development 
program : 


■  It  must  be  written  in  a  high-level  language  such  as 
Ada . 

•  It  must  allow  configuration  via  a  centralized 

controlling  unit. 

•  It  must  include  a  realistic  CPU  loading  capability. 

•  It  must  support  emulation  of  periodic  real-time 

processes  with  deadlines. 

•  It  must  include  emulation  for  transient  periodic 

real-time  processes. 

•  It  must  support  a  complex  message  passing  capability 
that  includes  performance  metrics . 

•  It  must  be  written  such  that  it  can  operate  in  a 

heterogeneous  environment . 

•  Each  experiment  must  be  repeatable. 

•  The  metrics  provided  must  be  useful  for  performance 
(i.e.,  loading)  tuning. 

•  In  general,  it  must  provide  the  ability  to  simulate 
processing  and  communication  workloads  on  multi¬ 
computer  networks . 
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C.  SYSTEM  DESIGN 

This  section  gives  the  reader  an  overview  of  the  AE 
system's  design,  and  insight  into  how  that  design  meets  the 
requirements  of  the  AE  project.  First,  we  will  start  by 
looking  at  how  the  AE  system  operates,  and  then  dissect  an 
AE  unit,  with  a  detailed  description  of  its  components. 
Figure  1  illustrates  a  prototypical  AE  system:  one  or  more 
AE  units  and  a  User  Interface  (UI)  .  The  most  important 
component  of  an  AE  system  is  the  AE  unit,  which  is  the 
emulator  engine  that  emulates  real-time  software 
components.).  This  diagram  is  similar  to  the  system  used 
for  experimentation  to  be  discussed  in  Chapter  IV. 

Each  AE  unit  in  Figure  1  can  be  located  on  any  computer 
and  operates  independently  of  the  other  AE  units,  so  all 
three  AE  units  in  Figure  1  could  be  located  on  the  same 
computer  or  three  different  ones.  To  change  the  experiment 
and  move  an  AE  unit  to  a  different  computer  only  requires 
changing  the  platform  where  each  AE  unit  is  started.  It  is 
worth  noting  that  the  example  in  Figure  1  does  not  account 
for  network  traffic  between  the  AE  units.  In  a  typical 
RTDS ,  network  traffic  would  be  present. 
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Figure  1  AE  System 


The  rest  of  this  chapter  describes  how  the  AE  system 
operates . 


1.  User  Interface  (UI) 

Emulation  using  the  Application  Emulator  (AE)  system 
consists  of  the  objects  shown  in  Figure  1:  Command  file. 
User  Interface  (UI)  and  one  or  more  AE  units.  The  UI  is  a 
multi-threaded  application  whose  main  task  is  to  control  the 
execution  of  a  group  of  AE  units  operating  as  a  RTDS.  The 
command  file  contains  a  list  of  commands  that  give  each  AE 
unit  its  resource  usage  profile.  The  command  file  drives 
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the  emulation  by  providing  the  resource  usage  information 
and,  in  addition,  provides  a  method  for  specifying  precise 
timing  of  resource  utilization. 

As  shown  in  Figure  1,  the  UI  plays  a  major  role  in  the 
execution  of  the  AE  system.  The  procedure  for  starting  all 
the  AE  units  and  the  UI  for  an  experiment  is  described 
below.  After  an  experiment  starts,  the  UI  reads  and 
processes  the  command  file.  Each  command  in  the  command 
file  contains  a  field,  which  specifies  a  particular  AE  unit 
by  name.  These  commands  contain  the  information  that 
controls  what  resource  and  the  amount  of  each  resource  each 
AE  unit  will  use.  Different  command  files  allow  the  AE 
system  to  emulate  an  entirely  different  system. 

The  UI  satisfies  several  of  the  high-level  requirements 
for  the  project.  It  provides  the  centralized  control  and, 
by  employing  the  command  file,  provides  for  repeatable 
experiments.  The  UI  also  allows  the  AE  system  to  be 
scaleable  (easily  supporting  many  AE  units  in  an 
experiment) ,  while  also  supporting  the  distributed 
architecture  requirement . 

The  startup  procedure  for  the  AE  system  can  aid  in 
understanding  how  it  operates.  Each  AE  unit  is  known  by  its 
name,  a  string  of  up  to  14  characters.  Use  of  names  to 
identify  and  establish  connections  in  the  network,  allows  an 
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AE  unit  to  be  executed  on  any  computer  system  on  the  Local 
Area  Network  (LAN) .  Starting  the  applications  manually  (the 
UI  and  the  AE  units)  on  several  different  computers  is  a 
multi-step  process.  To  save  time  and  reduce  mistakes,  an 
automated  startup  tool  that  greatly  simplified  the  otherwise 
labor-intensive  task  of  startup  was  borrowed  from  the 
DynBench  project.  The  startup  task  involves  one  UI  and  one 
or  more  AE  units. 

There  are  two  initialization  (i.e.,  command  line) 
parameters  of  interest  for  the  UI .  The  first  specifies  the 
mode  of  operation:  interactive  or  batch.  The  default  mode 
of  operation  is  batch  in  which  commands  are  processed  from  a 
file,  but  for  testing  and  flexibility,  an  interactive  mode 
is  available.  The  interactive  mode  includes  a  tool  that  can 
help  a  user  to  construct  the  lengthy  AE  commands .  The 
second  is  an  optional  parameter  that  specifies  the  number  of 
AE  units  that  will  be  part  of  the  experiment.  This 
parameter  instructs  the  UI  to  wait  until  that  number  of  AE 
units  has  connected  to  the  UI  (via  TCP/IP)  before  processing 
commands .  After  the  last  AE  unit  creates  a  connection  with 
the  UI,  it  starts  processing  the  command  file.  The  UI  also 
records  the  start  time,  which  is  used  for  Quality  of  Service 
(QoS)  data  and  for  commands  that  require  timing  (described 
below) . 


25 


There  are  four  types  of  commands  and  all  of  them 
support  an  optional  time  field  parameter.  This  option  is 
available  in  both  modes  of  operation  (interactive  and  batch) 
but  would  be  difficult  to  use  effectively  in  interactive 
mode,  since  interactively  constructed  AE  commands  are  likely 
to  execute  late.  Part  of  the  process  of  parsing  commands 
includes  checking  for  the  time  parameter.  If  the  time  field 
is  present,  then  the  UI  must  determine  if  the  command  is  to 
be  executed  immediately  or  later.  This  calculation  is  made 
by  comparing  the  elapsed  time  and  the  command's  time  field 
parameter.  Elapsed  time  is  defined  as  the  current  time 
minus  the  start  time.  If  the  current  command  needs  to  wait, 
then  the  UI  suspends  execution  until  the  time  field  and  the 
elapsed  time  are  the  same;  at  this  point  it  sends  the 
command  to  the  intended  AE  unit.  Because  the  command  file 
is  processed  from  top  to  bottom,  all  commands  that  follow 
one  with  the  time  parameter  specified  must  also  wait  until 
it  is  processed. 

An  AE  unit  supports  several  command  line  parameters, 
but,  for  normal  operation,  only  two  are  significant.  The 
first  one  specifies  the  AE  unit's  name.  The  second  one  is 
the  name  of  the  host  where  the  UI  is  executing.  This 
information  allows  this  AE  unit  to  make  a  network  connection 
with  the  UI .  The  mapping  of  an  AE  unit  name  to  IP  address 
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takes  place  at  run  time  and  as  stated  above  allows  AE  units 
to  be  executed  on  different  computers  for  different 
experiments  while  using  the  same  command  file.  The  name 
information  is  stored  in  the  connection  table,  which  is 
described  in  detail  in  Section  III.C.3,h).  The  connection 
table  is  replicated  on  the  UI  and  at  each  AE  unit.  This 
feature  allows  an  AE  unit  to  be  easily  extended  so  that  it 
can  be  moved  during  runtime  (this  extension  is  not  yet 
implemented).  The  table  includes  enough  information  (i.e., 
AE  unit  names  and  IP  addresses)  so  that  each  AE  unit  can 
create  a  network  connection  with  any  other  operating  AE 
unit . 

In  summary,  the  UI  plays  a  major  role  in  the  overall 
operation  of  an  experiment.  It  can  be  used  to  synchronize 
the  startup  process,  and,  because  it  records  the  start  time, 
it  also  allows  for  the  precise  timing  of  individual 
commands.  The  UI's  architecture  and  implementation  allow 
centralized  control.  Centralized  control  and  the  naming 
feature  allows  the  individual  AE  units  to  be  located 
anywhere.  The  UI  ends  an  experiment  when  it  encounters  the 
"stop  all"  command.  It  forwards  the  command  to  all  the 
participating  AE  units,  informing  them  to  perform  a  normal 
shutdown . 
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2. 


AE  Commands 


The  AE  supports  four  types  of  commands.  Each  command 
in  the  command  file  is  one  of  the  following  types: 

•  CPU  command, 

•  network  command, 

•  memory  command  or 

•  control . 

The  first  three  command  types  are  used  to  specify  resource 
loading  for  a  particular  resource.  The  command  structure 
developed  for  the  AE  system  is  shown  in  Appendix  B  and  an 
example  is  included  in  Appendix  A.  The  control  command  type 
is  used  for  shutting  down  the  system  after  an  experiment  or 
test  has  completed. 

3 .  AE  Unit 

Figure  2  depicts  all  of  the  major  internal  modules  of 
an  AE  unit  and  most  of  the  interactions  between  the  modules . 
All  the  shaded  objects  represent  a  process  thread.  As 
shown,  the  AE  is  a  multi-threaded,  complex  application.  Its 
components  include:  message  table,  connection  table,  network 
modules  (i.e.,  send  receive  and  a  general  networking 
module) ,  CPU  job  table,  monitoring,  message  processing,  CPU 
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loaders,  benchmarks,  controller  and  the  memory  loader.  The 
rest  of  this  section  describes  each  AE  component. 


Figure  2  AE  Unit  Block  Diagram 

a )  CPU  Loader 

Figure  3  is  a  diagram  showing  how  a  CPU  loader 
operates.  It  is  important  to  note  that,  for  real-time 
processes,  the  main  loop  will  operate  forever  (until  it  is 
stopped)  and  not  a  specified  number  of  times.  The  CPU 
Loader  along  with  the  workload  module  emulates  the  CPU 
workload  of  periodic,  aperiodic,  or  transient  periodic  real- 


29 


time  processes.  Each  AE  unit  has  the  ability  to  support 
multiple,  concurrent  executing  CPU  loaders. 


Figure  3  CPU  Loader  Functional  Diagram 

The  CPU  loader  module  has  several  features  that 
need  explanation.  The  call  to  the  workload  module  causes 
CPU  emulation  to  be  performed.  The  item  labeled  "action" 
drives  the  network  emulation  capability.  The  term  "action" 
and  how  it  applies  to  network  emulation  is  described  below. 
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The  last  feature  is  the  repeat  count;  this  feature  allows 
each  instance  of  a  CPU  loader  module  to  have  specified  both 
a  period  and  a  repeat  count.  For  example,  if  a  CPU  loader 
was  defined  with  a  period  of  two  seconds,  a  repeat  count  of 
four  and  an  action  to  send  a  message.  It  would  operate  as 
follows,  the  main  loop  would  start  every  two  seconds.  The 
inner  loop  would  iterate  four  times  for  each  main  loop 
execution.  Each  time  the  inner  loop  executes  it  would  call 
the  workload  module  and,  because  an  action  is  defined,  it 
would  also  send  a  network  message.  Therefore,  every  two 
seconds  the  loader  would  call  the  workload  module  four  times 
while  also  sending  four  messages.  The  sequence  described 
above  is  also  shown  in  the  timing  diagram  shown  in  Figure  4. 
The  term  action  as  it  is  used  in  this  thesis,  is  defined  as, 
"linking  CPU  processing  to  the  loading  of  other  resources", 
such  as  sending  a  message  after  completing  a  defined 
workload.  In  the  general  case,  real-time  components 
complete  the  same  task  repeatedly  in  a  periodic  (e.g.,  every 
second)  nature.  The  repeat  parameter  allows  a  periodic  CPU 
Loader's  period  to  be  divided  into  segments  so  that  an 
action  can  occur  several  times  in  a  single  period  (as 
diagramed  in  Figure  3)  .  The  parameters  listed  below  are 
configurable  at  CPU  loader  initialization  time: 
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real-time  period  (in  milliseconds) , 


•  benchmark  (Whetstone  or  Dhrystone) , 

•  workload  data:  average,  and  distribution  parameters 

(e.g.,  distribution  parameters  can  be  mean  and 

variance) , 

•  action  and  action  probability  (for  example  the 

action  is  taken  60%  of  the  time) ,  and 

•  repeat  value  (allows  actions  to  occur  several  times 
in  a  single  period) . 

Each  CPU  loader  task  maintains  the  following 

Quality  of  Service  (QoS)  data: 

•  deadlines:  missed  and  met,  and 

•  message  end-to-end  timing  information. 

The  CPU  loader  modules  can  operate  in  either  a 
periodic  or  an  aperiodic  manner.  Figure  4  illustrates  a 
loader  module  as  seen  by  someone  tracing  its  execution 

through  a  single  period. 
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Figure  4  CPU  Loader  Time  Diagram 

Figure  4  shows  a  periodic  CPU  loader  operating 
with  a  repeat  value  of  four  and  an  action  to  send  a  message 
(note,  that  in  this  example  the  probability  to  transmit  a 
message  is  100%) .  Remember  that  real-time  applications 
operate  in  a  periodic  fashion,  wake  up,  process,  sleep,  wake 
up,  process  sleep,  etc.  Here,  periodic  means  that  the  start 
times  are  uniformly  spaced  in  time.  The  above  diagram  is  a 
snapshot  of  such  a  process,  and  moving  ahead,  or  back  in 
time  will  produce  a  similar  diagram  with  evenly  spaced  start 
times.  The  diagram  shows  the  loader  first  simulating  CPU 
usage  followed  by  the  transmission  of  a  message  (through  an 
action)  .  The  above  sequence  is  executed  four  times  in  the 
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diagram,  at  which  time  the  loader  has  finished  its  CPU  and 
network  emulation  for  the  execution  cycle.  It  will  then 
suspend  execution  until  the  next  cycle  is  due  to  start. 

When  a  CPU  Loader  starts  a  new  execution  cycle,  it 
first  calculates  the  CPU  workload  using  the  average  and 
statistical  distribution  data.  The  data  used  in  the 
calculation  are  contained  in  the  AE  command  that  describes 
the  CPU  loader.  Workloads  can  be  described  as  normal, 
uniform  or  exponential  statistical  distributions.  Next,  a 
time  stamp  is  recorded  to  allow  for  QoS  measurements.  The 
workload  information  is  then  sent  to  the  benchmark  module  to 
emulate  CPU  loading.  As  an  example,  this  module  might  call 
the  Whetstone  Benchmark  to  simulate  CPU  loading,  or  workload 
as  it  is  referenced  in  this  thesis.  Each  action  has  an 
associated  probability  (0%  to  100%)  that  is  checked  before 
the  action  is  executed.  So,  if  an  action  is  defined  and  if 
the  probability  test  passes,  a  call  to  the  "network  send" 
(sending  a  message  is  the  only  implemented  action)  module  is 
made  with  the  information  needed  to  construct  the  size  and 
type  message  being  sent.  If  a  repeat  value  is  set,  then  the 
process  described  above  is  repeated  for  the  specified  number 
of  times.  Finally,  the  next  start  time  is  calculated.  If 
the  next  scheduled  start  time  has  passed,  then  a  deadline 
was  missed.  The  loader  records  the  event  (deadline  missed) , 
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and  starts  the  next  iteration.  If  the  deadline  was  met,  the 
loader  will  issue  a  sleep  command  to  consume  the  remaining 
time . 


b)  Workload  Module 

The  main  function  of  the  workload  module  is  to 
emulate  an  application's  CPU  resource  utilization.  To 
accomplish  this  task  the  AE  uses  a  list  of  commonly 
available  benchmark  programs,  which  provide  a  synthetic 
workload.  The  list  of  benchmarks  supported  includes  a  small 
Whetstone  [CURN76]  and  a  Dhrystone  [DHRY84]  benchmark. 
These  two  benchmarks  were  selected  because  they  represent 
computationally  intensive  workloads  and  the  class  of 
software  being  emulated  (real-time  distributed)  normally  can 
be  characterized  as  having  the  same  characteristics.  For 
completeness  and  flexibility,  the  design  and  implementation 
of  the  AE  allows  additional  benchmarks  to  be  easily  added  to 
the  existing  set. 
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c)  Ne tworking 


The  AE  system  has  been  developed  to  emulate 
existing  and/or  planned  RTDS .  In  that  environment,  some 
applications  only  process  messages;  their  workload  is  a 
function  of  the  number  and  type  of  messages  that  they 
receive.  A  message  received  by  an  AE  unit  can  contain 
workload  information.  Details  of  message  content  and  how 
messages  are  processed  by  an  AE  unit  are  fully  explained  in 
a  subsequent  section  (Message  Processing  III.C.3.j). 


d)  AE  Messages  (Network  Loading) 

An  AE  message  consists  of  several  data  fields, 
name  fields  (i.e.,  AE  names),  and  QoS  fields  which  contain 
timing  information.  The  name  fields  contain  the  originator 
and  all  the  receivers  of  that  message.  The  names  define  the 
arcs  that  a  message  takes  through  an  AE  topology  and 
determine  what  communication  connections  are  required  to 
support  that  message.  Four  different  message  path  types  are 
supported  by  the  AE  project:  simple,  fan  out,  pipeline  and 
circular  pipeline.  They  are  illustrated  in  Figure  5  and 
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were  selected  because  they  encompass  most  of  the 
communication  functionality  found  in  modern  large  real-time 
distributed  systems  today.  The  more  complicated  message 
types  (pipeline,  fan  out  and  circular  pipeline)  can  have  up 
to  five  receivers.  Five  was  selected  because  it  was  large 
enough  to  allow  the  AE  to  emulate  the  most  complex  message 
passing  used  in  the  HiPer-D  prototype.  A  larger  number  was 
not  selected  because  each  message  transmitted  between  AE 
units  carries  the  entire  data  structure  required  to  support 
all  the  features  of  the  networking  subsystem.  The  overhead 
of  supporting  up  to  five  receivers,  adds  300  bytes  to  each 
message.  A  larger  number  would  have  increased  the  overhead 
■of  the  AE . 

All  commands  (from  the  command  file)  are  processed 
through  the  UI  and  then  passed  to  an  AE  unit  as  described  in 
the  UI  Section  III.C.l.  Message  commands  require  an 
additional  parsing  step  by  the  UI  to  decode  the  message  type 
and  to  extract  all  sender  and  receiver  information.  The 
sender  and  receiver  information  is  then  sent  to  the  affected 
AE  units  to  inform  them  of  the  required  network  connections. 
The  following  parameters  are  set  when  defining  a  message 
(when  creating  an  AE  network  command) : 

•  message  type:  simple,  fan  out,  pipeline  or  circular 
pipeline 
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•  message  information:  size  and  message  size 

statistical  distribution  parameters  (e.g.,  size  mean 
and  variance) 

•  protocol  (UDP,  TCP)  and  port  number 

•  the  number  of  receivers  and  their  names 

•  unique  workload  information  for  each  receiver  of 
this  message  including  which  benchmark  to  use  for 
CPU  emulation. 


Sinple: 


Fanout: 


Pipeline: 

Circular 

Pipeline: 


Figure  5  Message  Paths  Supported 
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e )  Memory 

The  memory  emulation  capability  provided  by  the 
memory  module  uses  a  rudimentary  approach.  The  minimum 
memory  consumed  by  an  AE  unit  is  approximately  one  megabyte 
of  memory.  Memory  usage  is  emulated  by  allowing  an  AE  unit 
to  expand  its  total  memory  usage .  There  are  two  commands 
for  memory  emulation:  one  that  adds  to  the  current  size  of 
an  AE  emulator  and  an  other  that  decreases  the  emulator's 
size  (this  command  must  be  preceded  by  a  command  that 
increases  the  size) .  Reduction  in  memory  size  cannot  go 
below  the  actual  size  needed  for  the  AE  unit  itself.  For 
example,  if  a  particular  AE  unit  was  emulating  an 
application  that  has  a  run-time  size  of  3.5  megabytes,  then 
the  AE  would  need  to  add  2.5  megabytes  to  its  memory 
allocation  to  use  the  same  amount  of  memory.  The  AE 
emulates  the  application  memory  footprint  and  not  its  memory 
access . 

Memory  is  merely  allocated  is  not  used  or  accessed 
in  any  manner  by  an  AE  unit.  Normally  applications  allocate 
memory  for  a  reason,  and  they  normally  use  that  memory  for 
code  or  data. 
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f)  Message  Table 


AE  units  store  network  messages  in  the  message 
table  (see  Figure  2)  .  When  the  UI  processes  a  network 
message,  the  UI  sends  a  copy  of  that  message  to  the  message 
originator  (i.e.,  the  AE  unit  that  will  initiate  the  sending 
of  that  message)  .  Remember  that  an  AE  system  can,  in 
theory,  support  a  very  large  number  of  messages,  and  the 
discussion  below  describes  a  single  message.  All  messages 
are  static  in  that  they  always  start  from  the  same  AE  unit 
and  traverse  the  same  ordered  set  of  AE  units.  The 
originating  AE  unit  receives  a  copy  of  the  message  from  the 
UI  via  the  controller  as  shown  in  Figure  2 .  That  AE  unit 
then  inserts  that  message  into  its  message  table.  The 
actual  transmission  of  the  message  requires  the  send  module 
(see  Figure  2)  to  obtain  a  copy  of  the  message  from  the 
message  table. 

Figure  6  shows  the  data  structure  common  to  all  AE 
messages.  At  the  top  of  the  diagram  are  the  fields  that 
define  the  number  of  receivers  and  the  type  of  the  message 
(see  Figure  5  for  a  full  list  of  types)  .  This  is  followed 
by  the  workload  data  structure.  The  workload  data  structure 
contains  the  topology  information  (contained  in  the  AE  Name 
field)  and  the  workload  information  for  each  receiver  of  the 
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message.  The  Action  data  structure  at  the  bottom  of  Figure 
6  is  optional.  That  field  defines  an  action  (where  "no 
action"  is  a  valid  option)  and  the  probability  associated 
with  actually  executing  the  action. 


Workload  data 
structure 


Action  data 
structure 


Figure  6  AE  Message  Fields 


g)  CPU  Job  Table 

Periodic  and  aperiodic  tasks  are  emulated  using 
CPU  loader  jobs.  CPU  commands  take  the  same  path  as  network 
commands,  proceeding  from  the  command  file,  through  the  UI 
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to  the  appropriate  AE  unit .  Most  CPU  commands  become  CPU 
loader  jobs  when  they  are  received,  but  some  are  started 
and/or  stopped  by  events  (i.e.,  transient  periodic 
processes) .  The  job  table  is  where  event  processing  obtains 
the  parameters  to  configure  and  start  a  CPU  Loader  process. 


h)  Connection  Table 

Each  AE  unit  maintains  its  list  of  active  network 
connections  with  other  AE  units  in  the  connection  table. 
New  connections  between  AE  units  are  created  only  when 
required  by  a  network  message  command.  For  example,  if  a 
new  message  is  defined  that  goes  from  the  AE  unit  named  "A" 
to  "B",  then  "A"  and  "B"  consult  their  connection  table 
looking  for  an  existing  connection  using  the  same  protocol. 
The  supported  protocols  are  TCP  and  UDP .  If  one  exists, 
then  no  action  is  required.  If,  on  the  other  hand,  a 
connection  does  not  exist,  then  a  new  one  is  created  and 
information  about  the  AE  name,  IP  address  and  the  network 
channel  number  is  inserted  in  both  parties'  tables. 

The  circular  pipeline  message  passing  construct 
(Figure  5)  was  added  late  in  the  development  process  of  the 
AE  system.  The  existing  networking  code  for  the  project 
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contained  a  few  weaknesses  and  a  bug  surfaced  when  the 
circular  pipeline  construct  was  added.  To  explain  the 
problems  requires  a  basic  knowledge  of  how  a  TCP  connection 
is  created.  This  will  be  outlined  below. 

TCP  is  a  connection-oriented  protocol.  For  two  AE 
units  to  establish  a  TCP  connection,  one  side  (the  server 
side)  must  create  a  socket  and  then  "listen"  on  that  socket 
for  connections.  Meanwhile  the  other  AE  unit  (the  client 
side)  must  also  create  a  socket  and  then  through  that  socket 
it  attempts  to  connect  to  the  server  side  (using  IP  address 
and  port  number) .  Timing  is  a  critical  aspect  in  the  above 
sequence  of  events.  For  example,  if  the  client  attempts  a 
connection  before  the  server  is  ready  and  listening,  then 
the  client's  connection  attempt  will  fail.  On  the  server 
side,  the  listener  will  wait  indefinitely  for  a  connection 
unless  special  socket  options  are  used  to  cause  a  listener 
to  time  out . 

The  earlier  code  for  the  AE  project  attempted  to 
deal  with  the  problems  listed  above  by  using  less  than  ideal 
solutions.  The  old  technique  used,  described  below,  gave 
the  server  side  of  a  network  connection  a  small  time 
advantage  over  the  client  side.  The  time  advantage  was 
provided  by  the  UI's  action  of  sending  server  side 
connection  requests  to  the  AE  units  before  the  corresponding 
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client  side  connection  requests.  For  each  arc  in  a 
message's  voyage  (see  Figure  5),  the  UI  would  send  two 
connection  requests  commands,  one  for  the  receive  side 
(server  side)  and  one  for  the  sending  side  (client  side) . 
For  example,  a  circular  pipeline  message  with  three  AE  units 
(i.e.,  A  ->  B  ->  C  and  back  to  A)  would  produce  six 
connection  request  messages  from  the  UI :  three  send  and 
three  receive.  Each  AE  unit  for  the  circular  pipeline  case 
just  described  would  receive  two  connection  requests:  a  send 
and  a  receive  request.  This  usually  worked  by  allowing  the 
server  side  "some"  extra  time  to  establish  its  socket  before 
the  client  side  attempted  to  complete  the  connection.  The 
advantage  of  starting  earlier  usually  solved  the  timing 
problem,  but  because  it  did  not  eliminate  the  timing  issue, 
the  code  occasionally  failed. 

The  circular  pipeline  was  added,  because,  without 
it  the  AE  system  did  not  easily  support  two  way 
communications .  The  most  common  form  of  communication  is 
two  applications  communicating.  For  example,  "A"  sends  a 
message  to  "B",  "B"  processes  the  data  and  sends  a  response 
back  to  "A"  .  The  circular  pipeline  made  this  (and  more 
complicated  communication  topologies  where  the  originator 
receives  a  response  back)  much  easier  to  construct. 
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The  addition  of  the  circular  pipeline  message¬ 
passing  construct  caused  a  deadlock  when  using  the  older 
network  code.  Each  AE  unit  processes  network  connection 
commands  serially,  and  because  the  UI  made  sure  server  side 
connections  were  processed  first,  all  AE  units  involved  in  a 
circular  pipeline  message  were  acting  as  servers  waiting  for 
a  client  connection.  However,  the  same  set  of  AE  units 
waiting  for  a  client  side  connection  were  the  ones  that 
needed  to  also  act  as  clients.  The  result  was  a  deadlock 
situation. 

Consider  the  following  example.  If  "A"  and  "B" 
are  involved  in  a  circular  pipeline  connection  they  will 
both  receive  two  connection  requests,  a  server  side  with  the 
other  AE  unit  and  a  client  side  with  the  other  AE  unit. 
They  first  execute  a  blocking  call  to  listen  for  a 
connection  (server  side)  and  then  wait.  If  the  listen 
finished,  they  would  next  execute  the  client  side  of  the 
connection  but  because  neither  is  acting  as  a  client  and  the 
server  side  will  wait  indefinitely.  The  result  is  a 
deadlock . 

After  this  problem  was  discovered,  the  entire 
networking  code  was  reevaluated.  The  changes  included: 
multiple  retries  on  client  side  connections  (including 
progressively  longer  times  between  retries) ,  server  side 
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timeouts  (if  a  client  never  connects,  the  server  side  will 
give  up)  and  multiple  threads  to  process  connection 
requests.  The  use  of  multiple  threads  allows  an  AE  unit  to 
service  client  and  server  side  connection  requests 
simultaneously.  This  fixed  the  deadlock  situation.  These 
changes  fixed  all  the  known  problems  with  the  networking 
code. 


i)  Network,  Send  and  Receive 

This  section  describes  the  three  modules  that 
allow  network  communication  between  AE  units.  The  three 
modules  are  grouped  together  because  of  their  interactions 
and  common  functionality,  but  they  are  distinct  software 
modules.  The  network  module  was  written  in  Ada95  (as  was 
the  rest  of  the  AE  system)  but  the  send  and  receive  modules 
were  written  in  the  C  programming  language  because  of  its 
flexibility  and  system  interfaces. 

The  network  module  controls  the  networking 
functionality  for  the  AE .  Its  main  functions  include: 
processing  of  new  connection  requests,  checking  for  new 
messages,  preparing  messages  for  transmission,  and  recording 


46 


data  metrics  on  messages.  Each  of  these  areas  is  discussed 
below. 

The  processing  of  new  connection  request  allows 
the  transmission  of  messages  between  AE  units.  When  an  AE 
unit  is  initialized,  it  first  creates  a  TCP/IP  connection 
with  the  UI .  The  UI  reads  the  command  file,  and  when  it 
processes  network  commands,  it  sends  connection  requests  to 
the  appropriate  AE  units.  The  networking  module  acts  on 
these  requests  and  creates  the  necessary  networking 
connections  between  other  AE  units.  When  an  AE  unit  has 
active  network  connections,  it  periodically  polls  those 
connections  to  check  for  the  arrival  of  messages.  The 
networking  module  maintains  a  list  of  active  connections  and 
periodically  calls  the  receive  module  (described  below)  to 
check  for  messages.  If  a  message  is  received,  the  network 
module  performs  the  following  actions : 

•  insert  a  time  stamp  into  the  message  (time 

received) 

•  increment  a  counter  recording  the  number  of 

messages  received 

•  add  the  byte  count  of  the  current  message  to  the 
total  byte  count  for  the  protocol  (i.e.,  TCP) 

The  send  module  receives  its  input  from  the 

network  module  (described  above) .  Its  job  is  to  package  the 
three  components  of  a  message  into  a  buffer,  and  pass  the 
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message  to  the  Operating  System  (OS)  for  transmission. 
Figure  7  diagrams  the  three  parts  of  a  message.  The  amount 
of  padding  is  the  total  size  of  the  message  minus  the  other 
two  parts:  header  and  AE  network  command  data  structure  (the 
data  structure  is  diagramed  in  Figure  6) . 


Total  Buffer 
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Figure  7  Message  Layout 

One  of  the  problems  associated  with  a  wide  range 
of  message  sizes  is  maintaining  buffers.  Message  lengths 
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are  dynamic,  and  it  is  possible  that  the  next  message  will 
be  larger  than  the  current  buffer.  The  send  routine 
maintains  separate  buffers  for  sending  UDP  and  TCP  messages. 
The  first  step  in  building  a  message  for  transmission  is 
testing  the  message  buffer's  size  against  the  input 
parameter  that  defines  the  current  message's  length.  If  the 
current  message  buffer  is  not  large  enough  to  hold  the 
current  message,  then  a  new  buffer  is  allocated  and  the 
existing  one  is  released.  To  minimize  memory  allocation/de- 
allocation,  (generally  considered  a  problem  due  to  memory 
fragmentation  recovery  processes  that  can  cause  real-time 
systems  to  miss  deadlines  in  a  real-time)  system,  the 
following  technique  is  employed.  The  new  buffer  is  five 
kilobytes  larger  than  the  current  message .  The  value  of 
five  kilobytes  was  arbitrarily  chosen.  Although  the 
increase  in  size  is  much  larger  than  needed  for  the  current 
message,  the  overhead  and  impact  of  memory  allocation  is 
minimized. 

A  message,  as  diagramed  in  Figure  7,  is 
constructed  from  the  top  down.  First  the  header,  a  sixteen- 
byte  field  is  built  and  copied  into  the  buffer.  The  message 
header  is  described  in  detail  below  in  the  receive  module 
section.  Next,  the  instructional  part  of  the  message  is 
copied  into  the  buffer  (this  is  the  information  that  comes 
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from  the  AE  command  via  the  command  file)  .  The  message 
padding  is  not  formally  placed  into  the  buffer.  Because  the 
system  call  to  send  a  message  requires  a  pointer  to  a  buffer 
and  the  message's  length  in  bytes,  the  padding  is  safely 
included  in  the  message  by  ensuring  the  buffer  is  larger 
than  the  message  size. 

The  last  module  covered  in  this  section  is  the 
receive  module.  It  is  the  receive  module's  job  to  undo  what 
the  send  module  built  up  and  then  to  return  the  AE  command 
data  structure  to  the  message  processing  module  (covered  in 
section  III.C.3.j). 

There  are  differences  between  the  communication 
protocols  TCP  and  UDP  that  require  the  receive  module  to 
treat  these  protocols  separately.  TCP  messages  are  received 
as  part  of  a  flow  of  information  that  spans  messages.  UDP 
messages,  on  the  other  hand  are  received  individually  with 
no  overarching  organization  imposed  upon  a  series  of 
messages.  In  between  the  sender  and  the  receiver,  the 
network  components  may  break  up  a  UDP  packet  into  separate 
IP  packets  but  the  receiving  side's  OS  will  deliver  the  same 
size  message  to  the  receiver. 

To  receive  UDP  messages,  the  receive  module  calls 
the  recvfrom  system  call.  One  of  the  parameters  to  the 
recvfrom  system  call  is  the  number  of  bytes  to  read.  If 
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that  number  of  bytes  is  less  than  the  entire  message,  then 
part  of  the  message  will  be  lost.  For  example,  if  we 
receive  a  500-byte  UDP  message  and  only  read  the  first  100 
bytes,  then  the  last  400  bytes  are  lost  and  cannot  be  read 
later.  This  feature  actually  helps  the  AE  receive  messages 
because  the  number  of  bytes  that  must  be  read  is  known 
(header  and  AE  command  data  structure)  and  the  remaining 
bytes  can  be  safely  dropped. 

When  receiving  TCP  messages,  the  receive  module 
needs  to  maintain  message  boundaries.  Here,  the  main 
problem  is  that  a  receiver  does  not  know  the  length  of  a 
message  before  receiving  it,  and,  unlike  UDP,  all  the  bytes 
of  a  message  must  be  read  before  the  module  can  process 
future  messages.  The  header,  introduced  above  solves  the 
problem  by  providing  the  message  size  to  the  receive  module. 
A  message  header  contains  the  following  information: 

•  message  size  in  bytes, 

•  message  type  and 

•  endian  field  (described  below) . 

The  receive  module  will  first  issue  a  read  to 
obtain  the  header  information,  it  can  then  calculate  how 
many  additional  bytes  of  information  must  be  read  to  fully 
receive  that  message.  Next,  it  will  read  the  instructional 
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part  of  the  message  into  a  data  structure  that  will  be 
returned  to  the  calling  procedure.  The  final  step  is  to 
read  the  remaining  bytes  of  the  message.  These  bytes  are 
formally  known  as  the  message  padding,  and  they  are  read  and 
discarded. 

The  network  receive  module  uses  a  unique  and  fair 
method  for  processing  messages  over  multiple  active  network 
channels.  The  fairest  way  to  process  messages  would  be  as 
they  were  received.  Unfortunately  most  operating  systems  do 
not  instruct  an  application  that  has  more  than  one  pending 
message  any  timing  information  on  those  messages.  Fair 
means  that  if  the  last  message  received  was  from  Connection 
Channel  Three,  and  now  the  AE  unit  has  two  messages  ready 
for  processing  (one  on  Channel  Three  and  one  on  Channel 
Five),  then  we  will  process  the  message  from  Channel  Five. 
The  implementation  uses  an  integer  to  remember  the  last 
active  channel .  When  more  than  one  channel  has  a  message 
ready  for  processing,  the  AE  uses  a  modular  counter  to 
select  the  next  message  for  processing.  that  is,  the  AE 
unit  will  choose  the  channel  numerically  higher  than  the 
last  one  selected  (the  selection  will  wrap  around  to  zero  if 
the  last  one  selected  is  numerically  the  highest  in  the 
set).  It  is  the  author's  observation  that  most 
communication  software,  using  the  select  system  call  will 
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favor  lower  number  (over  higher)  channels  when  two  or  more 
messages  are  ready  for  processing  simultaneously.  The 
technique  developed  for  the  AE  appears  to  be  unique. 

To  summarize,  the  network  modules  take  care  of 
many  issues  related  with  communication  over  networks  and 
allow  the  AE  system  to  emulate  complex  message  passing 
applications.  The  sending  modules  build  up  the  messages  for 
transmission.  The  receiving  module  processes  the  header 
information  to  1)  deal  with  endian3  issues,  2)  identify  the 
message  and,  3)  by  using  the  size  information,  safely 
receive  any  size  message.  After  a  message  is  received,  it 
is  returned  to  Message  Processing  for  further  processing. 


3  Endian  refers  to  one  of  the  many  data  compatibility  issues 
that  can  occur  when  computer  systems  from  different 
manufacturers  or  operating  systems  communicate  over  a 
network.  The  endian  problem  stems  from  the  fact  that  some 
data  types  are  stored  differently  on  different  computers. 

Big  endian  systems  store  the  most  significant  part  of  the 
number  of  some  data  types  first  (lower  address  value)  and 
little  endian  systems  store  the  values  in  a  reversed  manner 
[STEV98] .  The  endian  field  (borrowed  from  HiPer-D)  provides 
a  nice  method  for  a  receiver  to  quickly  determine  if  the 
message  received  needs  an  endian  conversion.  The  field 
contains  a  value  that  when  tested  informs  the  receiver  if 
the  message  requires  an  endian  conversion  or  is  fine  as 
received. 
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j )  Message  Processing 


The  input  to  this  module  is  the  output  from 
Network  receive  component  (section  III . C. 3 . i)  .  The  receive 
module  returns  known  data  types  (i.e.,  AE  network  commands) 
and  places  them  on  a  circular  queue.  The  message-processing 
thread  is  event-driven  and  if  no  messages  are  available  for 
processing  it  stays  in  a  blocked  state  (to  reduce  CPU 
usage)  .  If  the  queue  is  empty  then  the  message-processing 
module  remains  blocked.  The  event  of  adding  an  item  to  the 
queue  unblocks  the  message  processing  thread.  This  feature 
is  implemented  using  ADA95  protected  objects.  Protected 
objects  operate  provide  mutual  exclusion.  The  rest  of  this 
section  contains  a  description  of  how  messages  are  processed 
by  AE  units . 

The  design  decision  to  separate  message  reception 
from  message  processing  allows  the  receiving  thread  to 
efficiently  receive  pending  messages.  The  processing  of 
messages  can  be  time  consuming,  and  is  therefore  handled  by 
a  separate  thread.  The  following  steps  outline  what  each  AE 
unit  does  to  process  a  message: 

•  compute  and  complete  CPU  workload, 
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•  forward  the  message  if  necessary,  and 

•  execute  an  event  if  necessary. 

Each  message  contains  workload  information  for 
each  receiver  (workloads  are  uniquely  defined  for  each 
receiver  of  a  message) .  The  pipeline,  circular  pipeline  and 
fan-out  (Figure  5)  message  constructs  are  examples  of 
messages  that  can  have  several  receivers.  Workload 
information  is  defined  with  the  same  parameters  as  the  CPU 
loader,  and  therefore  it  is  described  as  a  statistical 
distribution.  The  workload  emulation  uses  the  same 
benchmark  module  as  the  CPU  loader  module.  The  workload  is 
used  to  simulate  the  work  involved  in  message  reception  and 
processing.  Next,  the  message  is  checked  to  determine  if  it 
should  be  forwarded.  A  pipeline  message  (Figure  5)  is  an 
example  of  a  message  that  some  AE  units  (B  and  C)  would  need 
to  process  and  then  forward.  If  this  AE  unit  is  the  last 
receiver  of  a  message,  then  an  optional  event  can  be 
included.  Events  can  be: 

•  start  a  CPU  loader  job, 

•  stop  a  CPU  loader  job,  or 

•  send  a  new  message. 
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All  events  have  an  associated  probability.  This 
gives  the  AE  system  the  ability  to  dynamically  alter  its  own 
behavior  and  fulfills  the  requirement  of  supporting 
transient  periodic  processes. 


k)  Controller 

The  Controller  provides  the  interfaces  between  the 
UI  and  the  internal  modules  of  an  AE  unit.  Its  major 
function  is  to  receive  commands  from  the  UI,  process  those 
commands  and  then  issue  the  commands  to  an  appropriate 
module  within  the  AE  unit.  Further,  the  controller  reports 
new  information  to  the  UI .  There  are  four  types  of  commands 
that  the  controller  has  to  process:  CPU,  memory,  message, 
and  shutdown.  Once  a  command  is  identified  (e.g.,  a  CPU 
command)  it  is  sent  to  the  appropriate  module  for 
processing.  A  CPU  loader  command,  for  example,  will  create 
a  new  CPU  loader  process.  Because  all  timing  issues  related 
to  commands  are  handled  by  the  UI,  the  controller  merely 
processes  commands  when  they  are  received. 
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D .  SUMMARY 

This  chapter  has  described  the  AE  system.  It  started 
with  a  high  level  view  of  the  AE  system  as  it  would  be  used 
as  an  emulation  tool  (Figure  1  AE  System) .  Next  that  system 
was  examined  at  a  component  (i.e.,  AE  unit,  see  Figure  2) 
and  at  a  sub-component  level.  At  the  sub-component  level, 
many  of  the  details  about  the  AE  unit  were  explained.  In 
addition,  some  of  the  problems  encountered  while  developing 
the  AE  system  were  also  discussed. 

The  next  chapter  will  present  results  from  a  series  of 
emulation  experiments  using  the  AE  system.  A  tactical 
modeling  tool  was  used  as  the  target  application  for  the 
emulation.  The  results  show  that  software  emulation  using 
the  AE  system  can  be  effective. 
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IV.  SYSTEM  EMULATION  AND  EXPERIMENTATION  RESULTS 


A.  INTRODUCTION 


This  chapter  describes  how  the  AE  system  can  be  used  to 
emulate  an  existing  software  system.  The  emulation  process 
has  three  major  steps.  For  demonstration  purposes,  an 
example  that  emulates  a  system  from  Teledyne  Brown  called 
EADSIM  [EADOO] ,  is  used.  Before  the  steps  are  described,  an 
overview  of  EADSIM  is  presented.  The  final  section  of  this 
chapter  describes  the  work  required  to  validate  the  AE 
system's  accuracy  in  emulating  a  real  system. 


B .  EADSIM 

■  Extend  Air  Defense  Simulation  (EADSIM)  is  a  warfare 
modeling  program.  EADSIM  is  widely  used  to  model  battle 
scenarios  as  an  aid  in  making  tactical  decisions.  It 
consists  of  four  modules:  C3I,  Detection,  Propagation  and 
Flight  Processing.  These  modules  operate  in  a  distributed 
fashion  and  thus  use  networking  protocols  to  communicate. 

One  of  the  four  modules  is  optional  (i.e., 
Propagation)  and  was  not  included  in  Porter's  [PORT99] 
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thesis  results,  which  are  used  as  input  to  the  AE's 
emulation  process.  EADSIM  supports  a  wide  range  of  tactical 
systems  that  can  be  included  in  a  model.  Battle  scenarios 
are  constructed  through  a  complex  iterative  process 
[PORT99] . 

For  the  purpose  of  this  thesis,  the  configuration  of, 
and  results  from,  EADSIM  are  interesting  but  not  necessary. 
Remember  that  the  AE  system  does  not  return  useful  results, 
but  rather  loads  the  system  as  if  a  useful  application  was 
running.  The  block  diagram  of  EADSIM  (Figure  8)  shows  the 
communication  paths  between  the  three  distributed  modules  of 
EADSIM.  The  resource  usage  results  from  Porter's  execution 
of  EADSIM  are  presented  at  the  end  of  the  chapter,  followed 
by  the  results  of  the  AE  systems'  emulation  experiment  using 
EADSIM. 


60 


Figure  8  EADSIM  Runtime  Block  Diagram 

C.  SYSTEM  EMULATION:  THE  THREE  STEP  PROCESS 

Starting  with  a  system  like  EADSIM,  and  automating  the 
steps  to  emulate  it  using  the  AE  has  always  been  a  desired 
feature  of  the  AE  project.  Figure  9  below,  shows  the  three- 
step  process  for  creating  an  emulation  from  an  existing 
system  using  the  AE  system.  For  reasons  listed  below,  the 
goal  of  automating  this  process  was  not  realized.  A  problem 
was  the  tools  (or  lack  of  tools)  needed  to  profile  an 
application's  components  to  produce  the  information  required 
to  construct  the  AE  command  file.  Also,  as  will  be  shown,  a 
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general  understanding  of  the  system  under  study  is  required 
and  cannot  be  obtained  from  the  profiling  tools. 


1  2  3 


Figure  9  diagrams  the  steps  involved  to  use  the  AE 
system  for  emulating  an  existing  system.  The  fourth  step, 
not  addressed  in  this  paper  is  a  response  loop,  which  allows 
the  emulation  process  to  be  tuned.  The  steps  shown  in 
Figure  9  will  be  described  for  emulating  the  EADSIM 
application  by  the  AE  system. 
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1. 


Step  One:  Gather  Resource  Usage  Data 


The  objective  of  this  step  is  to  gather  data  on  the  use 
of  system  resources  by  the  application.  As  shown  in  Figure 
9  the  wrapper  tool  was  used  for  component  profiling;  this 
profiling  tool  was  developed  by  Schnaidt  [SCHN98]  under  the 
MSHN  project  (MSHN  wrappers) .  The  MSHN  wrapper  tool 
operates  between  an  application  and  the  Operating  System 
(OS),  by  intercepting  system  calls.  Here  a  MSHN  wrapper  is 
a  low-overhead  component  that  usually  only  records  the 
parameters  to  a  system  call.  For  example,  a  network  send 
calls  the  OS  write  function,  the  MSHN  wrapper  interrupts  the 
write  function  and  records  the  number  of  bytes  followed  by  a 
call  to  the  underlying  OS  write  function.  All  applications 
(on  Unix)  call  the  exit  function  to  halt  normally.  As 
implemented  by  Schnaidt,  the  MSHN  wrapper  for  the  exit 
function  completes  its  job  by  obtaining  the  CPU  usage  data 
and  logging  all  the  resource  usage  data  collected. 

The  MSHN  wrappers  provide  network  and  CPU  usage  profile 
data.  All  the  data  provided  in  this  chapter  on  EADSIM  was 
compiled  by  N.  Wayne  Porter  and  was  obtained  from  his  thesis 
[PORT99]  .  These  data  are  then  used  as  input  into  Step  Two, 
outlined  below. 
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2.  Step  Two:  Using  Profile  Data  to  Construct  AE 
Commands 

Starting  with  the  profile  data  from  the  previous  step 
and  creating  AE  commands  that  accurately  emulate  a  system  is 
the  most  difficult  step  in  emulating  an  existing  software 
system.  Some  of  the  data  provided  by  the  MSHN  wrappers  can 
be  easily  converted  into  AE  commands,  while  other  data 
require  a  conversion  step.  This  section  will  cover  the 
details  of  converting  the  MSHN  wrapper  data  into  AE 
commands . 

The  network  data  are  in  a  format  that  maps  nicely  into 
AE  commands.  The  MSHN  wrappers  report  networking  data  in 
the  following  areas: 

•  total  number  of  messages  sent, 

•  total  number  of  messages  received, 

•  total  bytes  sent,  and 

•  total  bytes  received. 

The  conversion  from  the  above  format  into  AE  commands 
is  fairly  easy  because  both  systems  use  similar  units.  The 
AE  system  sends  messages  through  a  CPU  loader  module 1 s 
action.  Thus,  if  an  application  sends  4  messages  per  second 
and  the  CPU  loader  has  a  period  of  0.5  seconds,  then  in  each 
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period,  the  application  will  send  two  messages.  If  the 
number  of  messages  varies,  then  the  probability  (of 
executing  the  action)  and  the  repeat  value  (for  the  CPU 
Loader)  can  be  modified  to  obtain  the  desired  message  rate. 
The  AE  message  command  provides  the  AE  unit  with  the 
following  information:  size  in  bytes,  and  path  information. 
The  transmission  rate  of  a  message  is  related  to  a  CPU 
loading  parameter.  See  Section  III.C.2  for  a  more  complete 
description. 

The  CPU  workload  information  currently  provided  by  the 
MSHN  wrappers  does  not  easily  convert  into  a  format  that  can 
be  used  to  construct  AE  commands .  The  MSHN  wrappers  report 
CPU  utilization  in  seconds  (e.g.,  17.115  CPU  seconds)  for 
each  module.  An  AE  unit,  on  the  other  hand,  operates  in 
terms  of  Kilo-Whetstones  (1000  Whetstone  instructions)  or 
Kilo-Dhrystones .  The  mapping  between  these  units,  to  any 
degree  of  accuracy,  requires  executing  an  AE  unit  on  the 
same  computer  used  to  obtain  the  MSHN  wrapper  data. 

The  method  developed  to  address  this  uses  the  AE 
system' s  percentage  capability  to  output  the  number  of  Kilo- 
Whetstones  needed  to  utilize  the  CPU  at  100%  for  a  specified 
time.  To  specify  the  CPU  usage  in  that  manner  requires  a 
single  command  to  run  a  CPU  loader  module  for  one  iteration 
using  100%  of  the  CPU  for  the  amount  of  time  desired  (e.g.. 
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usage  =  100%,  number  of  iterations  =  1,  time  period  =  22.567 
seconds)  .  The  AE  unit  will  then  print  out  its  calculation 
for  the  number  of  Kilo-Whetstones  or  Kilo-Dhrystones  needed 
to  produce  the  desired  result .  Remember  that  these  are 
ideal  results.  If  you  actually  programmed  an  AE  unit  to 
execute  with  those  parameters  it  would  execute  that  many 
Kilo-Whetstones,  but  it  would  most  likely  not  finish  in  the 
time  specified. 

The  MSHN  wrapper  provides  the  number  of  CPU  seconds 
the  application  used  and  this  figure  is  used  as  the  number 
of  seconds  to  use  100%  of  the  CPU  in  order  to  emulate  the 
application's  CPU  usage.  The  number  of  Kilo-Whetstones 
returned  by  this  method  becomes  the  target  number  of  Kilo- 
Whetstones  for  that  module  to  execute  over  the  entire 
emulation. 

The  algorithm  used  by  an  AE  unit  to  calculate  the 
number  of  Kilo-Whetstones  needed  to  consume  a  percentage  of 
a  CPU  is  described  below.  When  an  AE  unit  is  asked  to  use  a 
percentage  of  the  CPU  (e.g.,  35%)  it  first  does  a  test 
designed  to  use  10  0%  of  the  CPU  for  a  short  period  of  time 
which  produces  a  usage  value  that  is  used  for  all  percentage 
calculations.  The  test  must  take  into  account  the  following 
assumptions  and  problems  associated  with  clock  granularity. 

•  For  short  time  periods  one  user  gets  100%  of  a  CPU. 
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•  Tests  that  are  short  have  problems  because  clock 
granularity  can  introduce  error. 

•  Testing  several  times  increases  the  odds  that  all 
tests  will  not  be  pre-empted  and  swapped. 

•  Longer  tests  reduce  the  clock  granularity  problem 
but  increase  the  preemption  problem. 


Time 


rp* 

Tic  Tic  Tic 

One  two  three 


_ Si 

Test  I 


Test  II 


_ ▼ 

Total  Time  =  0 

Total  Time  =  1 


Test  III 


Total  Time  =  1 


Figure  10  Time  Granularity  Example 


Figure  10  shows  how  clock  granularity  can  introduce 
problems  (or  error)  into  calculations.  The  basic  problem 
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stems  from  the  way  the  system  reports  time.  In  the  area 
between  Tic  One  and  Tic  Two,  the  system  will  report  the  same 
value  for  the  current  time.  Timed  events  that  operate  for 
short  periods,  relative  to  the  clock  granularity,  can  lead 
to  misleading  results.  Test  I  and  II  are  almost  the  same 
duration  but  return  values  that  would  create  different 
assumptions  about  their  performance.  Test  III  is  almost  two 
clock  tics  in  length  but  is  reported  as  being  one  clock 
tick.  In  this  example  test  II  is  the  only  one  where  the 
reported  time  is  close  to  the  actual  time. 

For  a  given  CPU  (i.e.,  computer),  we  wish  to  calculate 

the  number  of  KW  (Kilo-Whetstones)  that  can  be  executed  in 

one  second.  Two  of  the  values,  number  of  KW  (20,000),  and 

number  of  times  to  execute  the  test  (i.e.,  7)  were  selected 

while  accounting  for  the  problems  listed  above.  The  test 

recorded  the  start  time,  ts,  and  the  time  at  the  end,  te. 

xKW  MKW  „nr  MKW 

- = - =>  xKW  = - *  lsec  , 

lsec  (test)sec  (te  -  ts) 


Where 

xKW:  total  Kilo-Whetstones  needed  to  use  100%  of 

the  CPU  for  1  second 
ts-te:  elapsed  time 

MKW:  number  of  KW  used  for  the  test  (20,000) 


The  timing  performance  built  into  the  AE  system 
operates  at  the  millisecond  (ms)  time  interval.  As  stated 
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above,  an  AE  unit  can  be  programmed  to  consume  a  percentage 
of  the  CPU  (i.e.,  1-100%).  Therefore,  a  CPU  loading  task 

can  be  programmed  to  operate  with  a  period  of  x  ms  that  will 
consume  y  percentage  of  the  CPU.  The  formula  below 
calculates  the  number  of  Kilo-Whetstones  that  will  consume 
the  desired  percentage  of  the  CPU,  for  the  time  interval 
specified.  The  term  xKW  from  the  previous  formula  provides 
the  baseline  for  this  calculation. 

y KW  =  T*P*xKW  , 

Where 

yKW :  CPU  workload  in  KW 

T:  time  in  ms  (ex.  10ms  is  entered  as  .010) 

P:  percentage  (25%  is  entered  as  0.25) 

xKW:  The  value  of  KW  that  will  use  100%  of  the  CPU  for 
one  second 

A  short  example  will  illustrate  the  calculation.  If 
xKW  is  100  and  a  user  wants  a  periodic  CPU  load  that  uses 
50%  of  the  CPU  and  operates  with  period  of  half  a  second. 
It  is  easy  to  see  that  the  answer  should  be  25.  The  formula 
becomes:  0.5  (time)  *  0.5  (percent)  *  100  (xKW)  and  will 

yield  the  correct  answer. 

The  percentage  usage  option  for  the  AE  was  shown  to 
operate  as  designed.  The  Unix  top  command  was  used  to 
verify  the  percentage  usage,  because  it  reports  an 
application's  CPU  usage  as  a  percentage.  For  the 
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experiment,  the  CPU  load  was  set  to  4  0%,  and  a  time  period 
was  set  to  1  second.  The  top  command,  reported  the  AE  CPU 
usage  with  in  1%  of  the  desired  value.  The  results  validate 
that  the  method  and  parameters  selected  produce  the  desired 
CPU  loading. 

As  was  shown,  the  conversion  from  CPU  seconds  to  Kilo- 
Whetstones  is  possible  by  programming  an  AE  unit  to  use  100% 
CPU  utilization  for  the  length  of  time  reported  by  the  MSHN 
wrappers  for  CPU  usage.  The  data  for  the  conversion  of 
EADSIM  modules  from  CPU  seconds  to  Kilo-Whetstones  is 
contained  in  Table  1. 

When  an  application's  CPU  workload  is  expressed  in 
Kilo-Whetstones,  it  can  be  converted  into  the  command 
language  that  drives  the  AE  system.  It  should  be  noted  that 
this  method  is  simplified  from  the  normal  case.  Most 
applications  will  have  several  threads,  and  detailed 
information  about  each  one  may  be  necessary  to  fully  emulate 
the  application.  The  data  obtained  from  the  MSHN  wrappers 
does  not  give  any  details  about  how  many  threads  were 
operating  and  the  CPU  usage  of  each  thread. 

Although  the  CPU  percentage  usage  is  part  of  the  CPU 
emulation  capability  of  the  AE  system,  it  was  not  originally 
included  for  the  following  reason.  Applications  can  be 
profiled  by  percentage  CPU  usage  but  what  they  actually  do 
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is  complete  a  task  or  set  of  tasks.  If  an  application  is 
ported  to  a  different  computer  system  then  that  same 
application  may  finish  in  a  shorter  time  while  using  a 
smaller  percentage  of  the  total  CPU  capacity.  The  AE 
system's  CPU  emulation  is  centered  on  the  idea  that 
applications  complete  tasks  and  using  a  percentage  of  the 
CPU  does  not  allow  the  AE  to  illustrate  performance 
variations  in  different  computers  and  operating  systems. 
The  AE  uses  a  synthetic  workload  (Kilo-Whet stones)  to 
represent  (or  emulate)  actual  workload. 

Another  challenge  to  the  conversion  of  resource  data 
into  parameters  for  the  AE  command  language  was  that  the 
MSHN  wrappers  do  not  record  any  real-time  information. 
Real-time  information  must  be  obtained  through  an 
understanding  of  the  system  under  study. 

In  conclusion,  by  using  the  information  provided  by  the 
MSHN  wrappers,  and  a  working  knowledge  of  the  system  under 
study,  it  is  possible,  by  using  various  conversions,  to 
build  the  commands  for  the  emulation  (i.e.,  an  AE  command 
file) . 
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3. 


Step  Three:  Running  a  System  Emulation 


This  step  involves  taking  the  command  file  produced  in 
the  previous  step  and  executing  it  to  emulate  the  original 
system's  resource  usage  profile.  The  first  step  is  the 
process  of  starting  and  synchronizing  all  the  AE  units. 
When  all  the  components  (the  UI  and  all  the  AE  units)  are 
operational,  the  UI  begins  reading  and  processing  the 
command  file. 

This  section  contains  some  detailed  information  about 
the  startup  process  that  was  introduced  in  Chapter  3 .  Some 
of  the  details  in  this  section  review  that  material. 

Each  AE  unit  supports  several  command  line  parameters, 
but  only  two  of  them  play  a  role  in  the  distributed 
architecture  (the  others  allow  an  AE  unit  to  operate  as  a 
standalone  CPU  loader) .  The  first  parameter  defines  the  AE 
unit's  name,  and  the  second  parameter  contains  the  hostname 
of  the  system  where  the  UI  is  operating.  The  UI  also  has 
two  command  line  parameters  of  interest:  command- file,  which 
contains  the  commands  used  to  configure  the  AE  units;  and  an 
integer  parameter  which  informs  the  UI  as  to  how  many  AE 
units  are  participating  in  the  experiment.  The  command  file 
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contains  all  the  commands  that  give  each  AE  unit  its 
identity  (to  emulate  its  part  of  a  RTDS)  .  The  parameter 
informing  the  UI  of  the  number  of  AE  units  allows  the 
startup  process  to  take  an  indeterminate  amount  of  time  to 
complete.  The  UI  keeps  a  running  count  of  AE  units  and 
waits  until  they  all  have  an  active  connection  with  the  UI 
before  starting  an  emulation  experiment. 

Figure  11  diagrams  the  automation  for  running 
experiments.  There  are  two  levels  of  processing  above  the 
AE  system  level.  The  top  level  written  for  this  thesis 
starts  the  automated  startup  level  and,  after  AE  system 
completes,  this  script  will  copy  the  remote  data  files  into 
a  file  structure  defined  in  the  script's  configuration  file. 
The  automated  startup  script  (see  Figure  11) ,  borrowed  from 
the  DynBench  project,  starts  the  components  (the  AE  units 
and  the  UI)  and  supplies  them  with  their  command  line 
parameters.  Using  remote  authentication4  for  starting 
processes,  the  Startup  script  can  start  processes  on  any 
computer  system  on  the  LAN.  The  configuration  files  for  the 
two  tools  are  included  in  Appendix  C.  Figure  11  contains  a 
graphical  representation  of  the  tools. 

4  Remote  authentication  allows  user  and  system  pairs  to  be 
mutually  trusted,  and,  as  such,  can  execute  commands  without 
presenting  a  password  as  might  be  required  in  an  interactive 
session.  [UNIX97] . 
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Automated  Emulator  Execution  Diagram 


Top  Level:  Starts  Automated  level 
Copies  Log  and  Data 
files  to  local  system 


Start  Up:  Automated  startup, 

starts  all  AEs  and  the  U1 
with  command  line  arguments 


Emulation:  Emulator  runs,  produces 
Log  and  Data  files 


Figure  11  Automated  Emuation  Diagram 

Once  all  the  necessary  AE  units  for  an  experiment  have 
connected  with  the  UI ,  the  emulation  process  begins.  All  AE 
commands  have  an  optional  time  parameter,  which  is  based  on 
the  time  that  the  last  AE  unit  established  a  connection  with 
the  UI  (i.e.,  elapsed  time).  As  described  earlier,  the  UI 
processes  the  command  file,  and  then  issues  each  command  to 
the  appropriate  AE  unit.  Normally  the  last  command  in  the 
command  file,  is  the  stop_all  command,  which  instructs  all 
the  AE  units  to  perform  a  normal  shutdown.  Before  shutting 
down,  each  AE  unit  outputs  all  its  data  and  debugging 
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information  to  a  data  and  log  file.  The  data  file  contains 
the  following  data  relevant  to  QoS  considerations: 

•  network  usage  data 

-  number  of  messages  sent  and  received 

-  total  number  of  bytes  sent  and  received 

-  timing  information  on  each  message 

•  CPU  usage  data 

-  total  number  of  Kilo-Dhrystones  executed 

-  total  number  of  Kilo-Whetstones  executed 

•  deadline  information 

-  Each  CPU  loader  module  records  the  number  of 
deadlines  missed 

The  next  section  compares  the  information  obtained  from 
experimental  runs  of  EADSIM  (the  data  from  the  AE  system  is 
obtained  from  the  data  files)  with  the  target  numbers  for 
the  emulation  of  EADSIM. 

D.  EADSIM  WRAPPER  RESULTS 

The  data  in  Table  1  was  obtained  from  Porter's  [PORT99] 
thesis.  He  obtained  the  data  from  the  MSHN  wrappers  while 
executing  EADSIM  as  shown  in  Figure  8. 
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Table  1  EADSIM  Resource  Usage  Data 


EADSIM  Resource  Usage  Data 

C3I 

FP  User 

Detect 

User  CPU  time 

17.717 

17.125 

16.316 

System  CPU  time 

3 . 026 

3 . 196 

5.855 

Total  CPU  time 

20.743 

20.321 

22.171 

Wall  time 

94.5 

77.1 

93.3 

Bytes  written 

1,634,436 

1,029,378 

2,057,529 

Number  of  writes 

155,957 

741 

589 

The  three  data  columns  in  Table  1  are  labeled  by  EADSIM 
modules  (see  Figure  8)  .  As  shown  in  Table  1  the  wall  time 
(i.e.,  actual  execution  time)  is  much  longer  than  the  CPU 
usage  time.  It  is  important  to  note  that  EADSIM  is  not  a 
real-time  application,  but  is  similar  to  a  real-time 
application  in  that  its  operations  are  time  stepped 
[PORT99] .  The  importance  of  time  makes  sense  because  a 
battle  simulator  must  account  for  when  and  where  events 
happen.  Table  2,  contains  the  conversion  from  the  MSHN 
wrapper  CPU  data  into  Kilo-Whetstones.  The  Kilo-Whetstones 
numbers  are  the  target  CPU  usage  numbers  for  the  three 
components  in  the  emulation. 
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Table  2  CPU  Resource  Usage  Data  for  EADSIM 


EADSIM  CPU  Resource  Data  and  AE  Conversion  CPU  Data 

C3I 

FP 

Detect 

CPU  time 

20 . 743 

20.321 

22.171 

Kilo-Whetstones 

906,096 

887,662 

968,474 

Table  3,  contains  the  network  information  from  EADSIM. 
Included  is  a  new  row  that  shows  the  average  message  size 
transmitted  by  each  component . 


Table  3  Network  Usage  data  from  EADSIM 


EASDIM  Network  Data 

C3I 

FP 

Detect 

Number  of  writes 

155,957 

741 

589 

Total  bytes 

1,634,463 

1,029,378 

2,057,529 

Ave.  msg.  size 

10.5 

1389.2 

3493.3 

As  shown  in  Table  3,  the  average  message  size  sent  from 
C3I  was  between  10  and  11  bytes.  The  AE  system's  minimum 
message  size  is  approximately  300  bytes,  due  to  the  overhead 
introduced  by  the  complexity  of  the  AE  messaging.  The 
result  is  the  AE  cannot  emulate  small  messages.  A 
compromise  to  permit  emulation  of  C3l's  network  traffic  was 
to  lower  the  total  number  of  messages  while  increasing  the 
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size  of  the  average  message  to  a  value  that  the  AE  system 
could  support.  The  resulting  emulation  reasonably  matches 
the  number  of  bytes  sent  by  C3I,  but  not  the  number  of 
messages  sent. 

EADSIM' s  operation  is  time  stepped.  C3I  controls  the 
execution  by  issuing  commands,  which  include  timing 
information  to  the  other  modules  (i.e..  Detection  and  FP)  . 
When  the  other  modules  complete  the  workload  for  current 
time  step  they  send  information  back  to  C3I,  and  the  process 
repeats  until  complete. 

Using  the  AE  system  to  accurately  emulate  EADSIM  will 
require  the  same  master/slave  relationship.  Because  the 
execution  of  EADSIM  is  time  stepped,  the  two  processes  that 
are  slaves  (Detection  and  FP)  to  the  master  (C3I)  will 
receive  all  their  workload  via  AE  messages.  The  method  of 
emulation  was  intended  to  simulate  actual  operation,  where 
the  three  components  of  EADSIM  complete  one  time  step's  work 
and  then  wait  for  the  command  to  start  the  next  time  step. 

The  algorithm  used  to  construct  the  command  file  for 
emulating  EADSIM  is  described  below.  The  command  file  used 
for  the  emulation  is  included  in  Appendix  B.  The  wrappers 
provided  high-level  usage  information  about  the  three 
modules  of  EADSIM.  The  wrappers  did  not  provide  detailed 
information  about  EADSIM  execution  characteristics.  For 
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example,  it  was  not  possible  to  tell  whether  or  not  the  CPU 
usage  of  FP  occurred  evenly  over  the  execution  time  or  if 
it  had  periods  of  greater  usage  and  other  times  of  much 
lower  than  average  usage.  Without  detailed  usage 
information,  the  emulation  was  forced  to  assume  that  the  CPU 
usage  was  consistent  over  the  wall  clock  time  of  the 
execution  of  EADSIM  (94.5  seconds) .  EADSIM  is  driven  by  the 
C3I  process;  it  provides  the  timing,  through  commands  to  FP 
and  Detect.  Therefore,  the  emulation  will  focus  on  C3I 
process  and  use  the  same  architecture  to  drive  the  other  two 
modules .  The  easiest  way  to  construct  the  emulation  was  to 
treat  the  relationship  between  C3I  and  detect  separately 
from  the  relationship  between  C3I  and  FP.  Therefore,  the 
CPU  load  for  C3I  will  be  divided  in  to  two  CPU  Loader  tasks. 
One  CPU  Loader  task  sent  messages  to  FP  and  the  other  sent 
messages  to  Detect.  Both  messages  contained  the  workload 
information  necessary  to  emulate  CPU  usage  for  FP  and 
Detect .  Other  messages  that  are  part  of  the  EASDIM  system 
were  emulated  using  events.  Table  4,  lists  emulation  target 
values  and  experimental  results .  It  is  important  to 
remember  that  the  target  numbers  are  emulation  target 
numbers  and  are  not  MSHN  wrapper  results.  The  target 
numbers  were  established  through  the  conversion  process 
described  earlier.  Recall  also  that  C3l's  number  of 
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messages  sent  was  modified  to  account  for  a  shortcoming  in 
the  AE  system. 


Table  4  Target  and  Emulation  Results 


Target  Numbers  for  EADSiM  Emulation 

C3I 

FP  User 

Detect 

Kilo -Whetstones 

906 , 096 

887,662 

968,474 

Messages  sent 

5,448 

741 

589 

Messages  received 

1,330 

2,724 

2,724 

Bytes  sent 

1,634,463 

1,096,533 

2,057,529 

Bytes  received 

3,086,907 

817,232 

817,232 

Experimental  Results  From  AE  System  Emulation  (Averages) 

Kilo- Whet stones 

925,979 

887,365 

967,881 

Messages  sent 

5,441 

781 

566 

Messages  received 

1,345 

2,720 

2,720 

Bytes  sent 

1,719,738 

1,096,533 

1,984,406 

Bytes  received 

3,059,339 

816,082 

816,586 

Percentage  nrrcr  (absolute  value) 

Kilo -Whetstones 

2.19% 

0 . 03% 

0.06% 

Messages  sent 

0.13% 

5.40% 

Messages  received 

0 . 15% 

0.15% 

Bytes  sent 

5.22% 

0.00% 

3.55% 

Bytes  received 

0.89% 

0 . 14% 

0.08% 

Figure  12,  shows  how  the  three  emulation  steps,  and 
tables,  presented  in  this  section  fit  together.  The  data  in 
Figure  12  is  the  average  values  for  the  resource  loading. 
The  exception  is  the  target  numbers,  which  are  calculated 
from  the  resource  usage  data.  The  emulation  experiment 
shows  that  the  AE  system  can  be  used  to  emulate  existing 
systems,  and  that  it  can  produce  results  that  are  within  a 
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small  percentage  of  the  target  values.  It  is  important  to 
note  that  all  emulations  will  have  some  variation  from  the 
actual  resource  loading  of  the  system  being  emulated.  For 
this  experiment,  the  amount  of  variation  from  the  target 
numbers  as  shown  in  Table  4  was  small. 

The  results  presented  are  from  a  sample  size  of  103 
emulation  runs  of  EADSIM.  Appendix  E  shows  a  full 
spreadsheet  containing  the  data  collected  from  the  103 
experimental  executions  of  the  AE  system  emulating  EADSIM. 
As  can  be  seen  by  examining  the  data  in  Appendix  E,  the 
results  for  most  of  the  metrics  are  fairly  close  to  the 
average  for  all  runs.  The  data  in  Table  4  contains  the 
average  values  for  all  the  metrics  recorded. 
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I Eadsim  Applications  (Wrapper  Data) 


AE  Target  Numbers 

AE  Actual  Numbers 

C3I 

FP  User 

Detect 

C3! 

FP  User 

Detect 

CPU 

906,096 

887,662 

968,474 

CPU 

925,979 

887,365 

967,881 

Messages  sent 

5,448 

741 

589 

Messages  sent 

5,441 

781 

566 

Messages  Received 

1,330 

2,724 

2,724 

Messages  Received 

1,345 

2,720 

2,720 

Bytes  sent 

1,634,463 

1,096,533 

2,057,529 

Bytes  sent 

1,719,738 

1,096,533 

1,984,406 

Bytes  Received 

3,086,907 

817,232 

817,232 

Bytes  Received 

3,059,339 

816,072 

816,586 

Figure  12  Experimental  Results  Diagram  (Averages) 


Additionally,  the  results  show  that  the  AE  system 
operates  as  intended.  The  commands  were  carefully  written 
for  this  experiment,  but  if  the  AE  system  had  not  operated 
as  intended  then  the  results  would  have  showed  a  larger 
percentage  error  for  one  or  more  of  the  recorded  metrics. 
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E.  SUMMARY 

This  chapter  showed  how  an  existing  system  can  be 
profiled  by  starting  with  data  obtained  using  the  MSHN 
wrappers.  Further,  we  described  how  that  MSHN  resource 
usage  data  can  be  used  as  input  into  a  process  that  can 
build  all  the  necessary  configuration  files  for  an  emulation 
using  the  AE  system.  As  was  shown,  the  emulation  can  then 
be  executed  and  the  results  obtained  from  the  AE's  data 
files  can  be  compared  to  calculated  resource  usage  values. 
The  results  obtained  showed  that  the  AE  system  did 
accurately  emulate  EADSIM  resource  usage.  Adjustments  can 
be  made  to  compensate  for  the  AE  limitations,  for  example 
the  fact  that  the  smallest  size  of  an  AE  message  was  much 
larger  than  that  of  the  application  it  was  emulating. 
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V. 


DISCUSSION  AND  CONCLUSIONS 


A.  LESSONS  LEARNED 

Some  potential  customers  of  the  AE  were  not  comfortable 
with  the  Ada  programming  language  selection.  Ada  is 
perceived  to  be  a  government  mistake  and  therefore  most 
sites  do  not  have  the  expertise  or  compilers  to  support  Ada 
development.  The  choice  to  develop  the  AE  in  Ada95  was  a 
good  technical  decision  but  possibly  a  poor  one  for 
marketing  the  AE. 

Much  of  the  AE  was  developed  using  Object-Based 
programming  and  not  full  Object  Oriented  (00)  techniques. 
"Object-Based  usually  refers  to  objects  without  inheritance 
and  hence  without  polymorphism"  [OBJFAQ] .  If  the  AE  project 
was  designed  and  developed  using  full  00  features  then 
future  changes  could  easily  produce  new  and  powerful 
capabilities,  while  leaving  the  existing  functionality 
intact.  The  current  design  allows  for  change  but  does  not 
leave  the  old  functionality  intact.  A  full  00 
implementation  would  have  been  a  wise  decision. 
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B .  FUTURE  WORK 

A  number  of  additions  to  the  AE  would  increase  its 
emulation  functionality.  One  improvement  would  be  to 
provide  a  general  mechanism  to  allow  the  AE  to  send  and 
receive  messages  from  existing  systems.  Used  in  this 
manner,  the  AE  system  could  obtain  its  loading  from  an 
existing  system  or  be  used  to  drive  an  existing  system. 
While  this  capability  exists,  it  is  limited  and  must 
currently  be  customized  for  each  type  of  message.  A 
general-purpose  method  for  this  type  of  feature  would  be  of 
great  value  for  a  real-time  development  project. 

The  use  of  multicast  could  reduce  the  complexity  of  the 
UI  to  AE  unit  communication.  Using  multicast  for  the  Ul-to- 
AE  communication  would  eliminate  the  command  line  parameter 
to  the  AE  used  for  finding  the  UI .  This  feature  would  also 
help  the  implementation  of  migration  of  AE  units  while  an  AE 
system  is  operating  (one  of  the  features  not  yet 
implemented) . 
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Other  future  features  include  different  load  simulators 


for  many  of  the  other  resources  that  applications  utilize. 
The  following  list  contains  some  of  the  resources  that  would 
increase  the  emulation  capability  of  the  AE. 

■  File  access  (local  and  file  server) 

■  Display  subsystem 

■  Database 

C.  COMPARISON  WITH  RELATED  WORK 


This  section  contains  the  comparison  of  this  thesis  and 
other  projects  that  are  closely  related  with  the  AE  system. 

1 .  DynBench 

A  common  goal  for  the  AE  and  DynBench  projects  was  to 
provide  researchers  tools  with  which  to  emulate  the  HiPer-D 
system.  The  approaches  taken  by  the  two  efforts  were  vastly 
different.  DynBench'' s  approach  was  to  build  a  simplified 
version  of  HiPer-D,  making  it  a  specialized  solution  to  the 
problem.  The  AE  system  on  the  other  hand,  is  a  general- 
purpose  real-time  application  emulator,  and  because  HiPer-D 
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is  in  the  class  of  systems  that  the  AE  can  emulate,  it  too 
can  provide  an  emulation  solution.  The  primary  task  in 
creating  such  an  emulation  would  be  the  construction  of  the 
command  and  configuration  files. 

The  HiPer-D  team  plans  to  combine  the  DynBench  and  the 
AE  system  and  make  the  combined  system  available  to  other 
researchers .  Users  will  be  able  to  use  the  AE  and  DynBench 
either  in  combination  or  individually.  These  two  systems 
are  complementary.  The  AE  system  offers  users  a  wide  range 
of  configuration  options  while  DynBench  offers  users  a 
specific  and  well-tuned  HiPer-D  emulation  tool. 

2.  Carff  Emulator 

Carff's  emulator  has  many  interesting  characteristics. 
It  is  a  distributed,  portable  (developed  in  Java) ,  message 
passing  application  emulator.  It  contains  many  of  the  high 
level  features  found  in  the  AE  system,  but  its  code  size  is 
at  least  an  order  of  magnitude  smaller  than  that  of  the  AE. 

There  are  several  differences  between  the  two  systems. 
The  main  one  is  that  the  AE  is  a  real-time  emulator  and 
Carff's  is  a  user-level  application  emulator  (applications 
that  execute  a  task  and  finish)  .  The  message  passing 
subsystems  are  vastly  different;  the  AE  supports  a  complex 
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yet  flexible  message  passing  subsystem,  while  Carff's  only 
supports  point-to-point  messaging.  The  CPU  workload  of  the 
two  systems  is  similar;  both  offer  a  wide  range  of  options 
for  providing  CPU  loading  with  statistical  variation. 

By  using  the  programming  language  Java  that  abstracted 
out  the  details  of  networking,  the  Carff  emulator  enjoyed  a 
much  shorter  development  cycle  than  the  AE.  In  contrast, 
networking  is  at  the  heart  of  the  AE  project.  For  the  AE, 
networking  took  the  lion' s  share  of  the  development  time  and 
introduced  most  of  the  difficult  problems. 

3 .  Petri  Nets 

Petri  Nets  are  a  tool  that  allows  researchers  an 
indirect  method  for  studying  systems.  The  method  includes 
building  a  mathematical  model  of  the  system  under  study. 
This  model  is  then  studied  in  a  laboratory  setting.  This 
indirect  method  of  study  is  useful  when  the  actual  system  is 
difficult  to  study. 

The  AE  will  allow  modeling  through  emulation,  and,  as 
such,  will  allow  Petri  net-type  analysis  of  some  systems. 
Using  a  loose  definition,  the  HiPer-D  system  is  an  example 
of.  a  Petri  net  system.  In  this  case,  it  is  safer  and  easier 
to  develop  and  study  the  system  in  a  lab  before  fielding  it 
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on  a  ship,  where  lives  and  operations  will  depend  on  its 
functionality. 

4.  Hartstone 

The  Hartstone  benchmark  [HART89]  is  a  tool  that  can  be 
used  to  prototype  real-time  systems  and  is  mainly  used  for 
studying  real-time  system  performance.  There  are  many 
similarities  and  differences  between  the  use  of  the 
Hartstone  benchmark  system  and  the  AE  project. 

Starting  with  the  similarities,  both  systems  can 
support : 

■  Prototyping  of  real-time  systems, 

■  Sending  and  receiving  of  messages, 

■  Periodic  and  aperiodic  tasks  and 

■  Synthetic  workloads. 

The  differences  between  the  two  tools  are  numerous.  The 
Hartstone  benchmark  is  intended  to  operate  as  a  single 
system  that  will  return  a  performance  metric.  The  metric  is 
either  on  (the  system  has  met  all  its  real-time  deadlines) 
or  off  (the  system  missed  at  least  one  real-time  deadline) . 
The  AE,  on  the  other  hand,  was  intended  to  be  a  tool  that 
operates  concurrently  with  other  systems.  Its  main  purpose 


90 


is  to  allow  experiments  to  determine  the  effects  of  CPU 
loading  and  network  communication  on  the  total  system.  Many 
of  the  other  differences  stem  from  that  difference.  For 
example,  a  Hartstone  test  will  terminate  when  a  deadline  is 
missed.  The  AE  simple  records  the  event  and  keeps  on 
executing.  The  messaging  subsystem  in  the  AE  reflects  the 
recent  growth  in  distributed  systems  where  communication  is 
not  always  point-to-point.  It  allows  for  messages  that  span 
several  applications  and  further  records  the  time  it  takes 
that  message  to  traverse  its  path.  Another  big  difference 
between  the  methods  relates  to  workload,  the  Hartstone 
benchmark  defines  workload  in  terms  of  percentages  while  the 
AE  uses  an  actual  value  (i.e.,  kilo  whetstones)  as  well  as 
percentages . 

The  AE  implements  or  is  designed  to  support  many  of  the 
latest  developments  in  real-time  software.  For  example, 
application  migration  would  not  be  supported  by  the 
Hartstone  benchmark.  The  Hartstone  Benchmark  is  primarily 
for  embedded  real-time  systems  [HART90] .  It  would  be  almost 
impossible  to  meet  a  deadline  if  an  application  were  to 
migrate  during  a  period.  The  new  approach  is  to  allow,  a 
system  experiencing  problems  to  miss  some  deadlines  while  a 
controlling  application  (i.e.,  a  RMS)  carries  out  an  effort 
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to  return  the  system  to  full  functionality  as  quickly  as 
possible . 

It  would  not  be  difficult  to  convince  someone  that  a 
system  that  is  critical  for  an  airliner's  operation  should 
be  able  to  recover  from  an  event  such  as  a  PC  crash.  This 
is  an  example  of  a  "real-time  mission-critical  system  that 
must  respond  in  a  timely  manner  to  conditions  in  their 
environment"  [WELC98] .  The  recovery  process  might  merely 
require  that  applications  that  existed  on  the  crashed  system 
be  moved  to  a  different  computer.  Thus,  the  whole  system 
could  be  restored  to  full  operational  status.  In  this 
scenario,  the  crash  may  cause  some  short  term  problems,  but 
if  the  remedy  is  applied  before  total  control  of  the 
aircraft  is  lost,  then  the  safe  recovery  can  be  achieved. 
The  AE  is  a  tool  that  can  support  this  paradigm  and  the 
Hartstone  benchmark,  while  an  excellent  tool,  cannot  support 
this  form  of  system  survivability. 


D .  CONCLUSION 


Members  of  the  HiPer-D  development  team  saw  a  need  to 
develop  a  real-time  application  emulator  to  help  them 
evaluate  their  prototype  Real-Time  Distributed  System 
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(RTDS) .  In  order  to  be  useful,  the  system  would  need  to  be 
able  to  easily  emulate  a  wide  range  of  real-time 
applications.  Further,  the  resource  usage  of  these  emulated 
applications  would  have  to  be  programmable.  The  existing 
set  of  tools  available  for  real-time  emulation  did  not  meet 
their  requirements. 

Starting  with  a  need  and  a  set  of  requirements,  the  AE 
project  set  out  to  build  an  emulation  application  that  could 
emulate  RTDS.  The  main  resource  areas  of  emulation  were  CPU 
and  network  usage.  The  emulation  was  designed  not  only  to 
match  how  much  of  a  resource  an  application  used  but  also  to 
closely  match  when  that  resource  was  utilized.  The  final 
product,  as  was  demonstrated  through  the  EADSIM  example,  has 
enough  built-in  emulation  capability  and  control  to  emulate 
a  wide  range  of  distributed  applications  accurately. 

In  conclusion,  AE  system  is  a  tool  that,  in  some 
cases,  can  aid  developers  of  real-time  systems.  As  the 
world  becomes  more  dependent  on  computers  and  especially 
real-time  computer  systems  for  safe  functionality  (e.g., 
aircraft)  ,  the  need  for  tools  to  help  design  and  prototype 
future  systems  increases.  The  AE  project  fits  nicely  with 
the  other  existing  tools  presented  in  this  paper  and  as  such 
has  the  potential  to  aid  in  current  and  future  real-time 
development  projects. 
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APPENDIX  A:  AE  COMMAND  FILE  FOR  EADSIM  EMULATION 


Below  is  the  command  file  that  was  used  to  Emulate 
EADSIM.  The  first  10  seconds  are  used  to  create  the 
communication  channels.  For  example,  the  ":0: 0:2:0:"  means 
that  0  hours,  and  0  minutes,  and  2  seconds,  and  0  ms  after 
starting,  execute  this  command.  The  message  definitions  are 
all  separated  by  two  seconds.  While  this  is  not  necessary 
in  theory;  sometimes  the  AE  system  will  have  problems 
processing  several  network  commands  at  the  same  time.  The 
actual  emulation  process  is  started  10  seconds  after  the 
synchronization  occurs.  The  "all  done"  and  "turn  off"  occur 
well  after  the  wall  clock  time  for  EADSIM  (i.e.,  its  CPU 
loaders  should  have  executed  the  number  of  iterations 
programmed,  recorded  their  CPU  usage,  and  QoS  information 
and  exited)  .  The  "turn  off"  is  the  command  for  the  UI  to 
exit.  At  that  point,  the  network  code  will  report  its  QoS 
data.  That  last  step  taken  is  to  write  and  close  the  data 
files.  None  of  this  is  shown  but  is  part  of  the  normal 
shutdown  process  for  an  AE  unit. 


#  Simple  C3I  : :  TO  : :  FP  and  Detect 

# 

0:0:2 :0:c3i  network  def ine_message  TCP  16081  300  normal  12  1  simple  fp 
326  normal  15  wheat  send_a_msg  3  27 

c3i  network  def ine_message  TCP  16082  300  normal  12  2  simple  detect  356 
normal  17  wheat  send_a_msg  4  22 
# 

#  simple  FP  : :  TO  ::  C3I, 

# 

0 : 0 : 4 : 0 : fp  network  def ine_mes sage  TCP  16083  1389  normal  46  3  simple  c3i 
0  normal  0  wheat  none 
# 

#  Simple  Detect  : :  TO  : :  C3I 

# 

0 : 0 :6 : 0 :detect  network  def ine_mes sage  TCP  16084  3494  normal  116  4  simple 
c3i  0  normal  0  wheat  none 
# 

0:0:10 :0:c3i  cpu  cpu_cmd  a_de  1  true  584  wheat  actual  167  normal  7  17 
true  send_msg  2  100  160 

0 : 0 :10 : 200 : c3i  cpu  cpu_cmd  a_fp  1  true  584  wheat  actual  167  normal  7  17 
true  send_msg  l  100  160 
# 

# 

0  :  0 : 110 : 0 :all  done 
0  :  0 :112 : 0 :turn  off 
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APPENDIX  B:  AE  COMMAND  STRUCTURE 


Appendix  B  contains  the  command  structure  for  the  AE 
system.  The  diagram  below  reads  from  the  top  to  the  bottom 
and  spans  the  next  few  pages .  At  each  location  in  a 
command,  where  the  value  in  that  field  will  cause  a  branch 
in  the  command  the  graph  also  has  a  branch  and  the  arcs  are 
labeled  with  the  choices  for  that  entry.  Because  some  of 
the  commands  are  quite  long,  the  diagram  is  continued  on  the 
following  pages  (Network  and  CPU) .  When  a  valid  entry  for  a 
command  has  only  a  few  choices,  they  are  placed  inside 
parentheses. 


This  diagram  is  a  continuation  from  the  previous  page 
and  reads  from  top  to  bottom.  It  shows  the  rest  of  the  AE 
CPU  commands.  Note,  the  last  entry  is  the  loader's  duration 
in  time  periods  (i.e.,  how  many  time  periods) .  A  loader  can 
operate  for  a  fixed  number  of  time  periods  (e.g.,  500)  or, 
forever,  as  would  be  the  case  for  most  real-time  process.  A 
value  of  zero  is  entered  when  the  loader  should  run  forever. 
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This  diagram  shows  the  AE  network  command  structure. 

Note,  the  number  of  receivers  is  a  number  from  one  to  five. 
The  loop  in  the  center  is  where  the  different  workload 
values  for  each  receiver  of  a  message  is  configured.  The 
bottom  of  the  diagram  illustrates  how  events  are  configured. 
Note  that  None  is  a  valid  event  (i.e.,  no  event) . 
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APPENDIX  C:  AUTOMATED  EMULATION  CONFIGURATION  FILES 


The  configuration  files  for  the  two  automated  startup 
tools  are  shown  below.  The  top  diagram  contains  the 
configuration  of  the  tool  that  starts  the  other  startup 
tool.  When  the  AE  system  finishes  this  tool  will  copy  the 
data  files  back  to  the  current  computer  system. 

The  lower  diagram  is  the  configuration  file  for  the  tool 
that  starts  the  AE  system.  It  needs:  the  path  name  of  the 
program  (i.e.,  AE  unit  and  UI) ,  command  line  parameters  and 
the  system  name  where  it  should  be  run. 


Load_sim_batch  Example  file  (Eadsim  batch) 

/^Mr  "N 

Eadsimjresults 
name 
c3i  alphel 
fp  alphe2 
detect  alphe3 
cmds 

Eadsim  Eadsim.start 
done 


Start  configuration  file  Example  Eadsim.  Start  (DynBench  tool) 


r 


tdrake;/home/usr/tdrake/LS/;ui.solaris2. 6.exe  file  Eadsim.cmd  3;alphe3; 
sleep  3 
# 


tdrake;/home/usr/tdrake/LS/;load_sim.solaris2.6.exe  name:c3i  ui:alphe3;alphel ; 
tdrake;/home/usr/tdrake/LS/;load_sim.solaris2.6.exe  name:fp  ui:alphe3;alphe2; 
tdrake;/home/usr/tdrake/LS/;load_sim.solaris2.6.exe  name:detect  ui:alphe3;alphe3; 
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APPENDIX  D:  LIST  OF  ACRONYMS 


AE  Application  Emulator 

AH  Hartstone  Benchmark  Standalone  test 

API  Application  Programmer  Interface 

ASCII  American  Standard  Code  for  Information  Interchange 

C3I  Command,  Control,  Communication  &  Intelligence 

C  The  C  Programming  Language 

C++  C  "plus  plus"  Programming  Language 

COTS  Commercial  Off  The  Shelf 

CPU  Central  Processing  Unit 

Detect  Detection  Process  (part  of  EADSIM) 

EADSIM  Extended  Air  Defense  Simulator 

FAQ  Frequently  Asked  Question 

FP  Flight  Processing  (part  of  EADSIM) 

IP  Internet  Protocol 

LAN  Local  Area  Network 

MSHN  Management  Systems 

NSWC  Naval  Surface  Warfare  Center 

00  Object  Oriented 

OS  Operating  System, 

PC  Personal  Computer 

PH  Hartstone  Benchmark  Standalone  test  (one  of  five  defined) 

PN  Hartstone  Benchmark  Standalone  test  (one  of  five  defined) 

QoS  Quality  of  Service 

RMS  Resource  Management  System 

RTDS  Real-Time  Distributed  System 

SA  Hartstone  Benchmark  Standalone  test  (one  of  five  defined) 

SH  Hartstone  Benchmark  Standalone  test  (one  of  five  defined) 

TCP  Transmission  Control  Protocol 

UDP  User  Datagram  Protocol 

UI  User  Interface 

WCS  Weapon  Control  System 
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APPENDIX  E:  DATA  FROM  A  SERIES  OF  EADSIM  EMULATIONS 


C3i  Data 


Sample  Time 
Sec. 


Work 

Kilo- 

Whetstones 


Messages  Messages 
Sent  Received 


Bytes 

Sent 


92.96 

910,744 

5,440 

92.94 

910,559 

5,440 

92.93 

911,995 

5,440 

■■EESEESJ 

HI 

911,405 

5,440 

92.93 

■HSBsl 

■HEESslI 

92.93 

909,406 

5,440 

92.93 

911,627 

5,440 

92.93 

912,939 

5,440 

92.93 

911,339 

5,440 

92.93 

909,341 

5,440 

92.93 

910,243" 

5,440 

HESS 

913,600 

5,440 

92.93 

■H 

5,440 

92.94 

910,333 

5,440 

92.93 

912,320 

5,440 

■ESS 

907,315 

5,440 

92.93 

908,027 

5,440 

92.93 

910,554 

5,440 

92.93 

906,189 

5,440 

92.93 

907,806 

5,440 

92.93 

912,677 

5,440 

92.93 

911,514 

5,440 

92.94 

909,408 

5,440 

92.93 

■n 

5,440 

92.93 

904,127 

5,440 

92.94 

907,137 

5,440 

92.93 

909,096 

5,440 

92.93 

909,624 

5,440 

92.93 

910,363 

5,440 

92.92 

908,086 

5,440 

92.93 

908,660 

5", 440 

92.92 

■EEEESI 

92.93 

908,524 

5,440 

1,363 

1 ,719,865 

1,331 

1,720,289 

1,339 

1,718,443 

1,314 

1,717,910 

1,404 

1,718,382 

1,385 

1,719,194 

1,379 

1,719,091 

1,365 

1,719,022 

1,380 

1,719,926 

1,355 

1,720,314 

1,397 

1,718,317 

1,375 

1,718,660 

1,377 

1,718,180 

1,341 

1,719,018 

1,385 

1,717,953 

1,298 

■MfcM.-l.-ll 

1,377 

1,719,910 

1,313 

1,719,558 

m 

1,317 

1,721,352 

1,370  1,718,171 


HI 


mm\ 


1,360  1,718,029 


1,3481  1,716,8331 


Bytes 

Received 

3,177,358 

3,131,094 


3,150,189 


3,086,010 


3,274,533 


3,240,254 


3,273,636 


3,155,483 


3,153,989 


mam 


3,185,017 


3,203,098 


3,115,931 


3,287,455 


3,054,718 


3,202,327 


3,033,111 


3,225,880 


3,236,355 


3,037,212 


3,241,987 


3,169,059 


3,187,086 


3,189,261 


3,186,036 


3,110,805 


3,159,311 


3,090,926 


105 


92.94 

911,536 

5,440 

92.93 

911,266 

■KiE2!]| 

92.93 

92.93 

911,930 

5,440 

92.93 

908,068 

5,440 

92.93 

907,018 

5,440 

92.93 

910,926 

5,440 

92.93 

■■EKESI 

92.93 

908,613 

5,440 

92.93 

912,559 

5,440 

92.93 

908,074 

5,440 

92.93 

907,023 

■■EES 

92.93 

907,188 

■BSI 

92.93 

906,722 

5,440 

92.93 

909,927 

5,440 

92.93 

911,130 

5,440 

92.93 

908,042 

5,440 

92.93 

910,476 

5,440 

92.93 

909,012 

5,440 

92.93 

905,559 

5,440 

92.93 

912,566 

5,440 

92.93 

■■SSI 

92.93 

■■ssi 

92.93 

909,705 

5,440 

92.93 

912,955 

5,440 

92.93 

909,801 

■■ssi 

92.92 

5,440 

92.92 

910,440 

5,440 

92.93 

heess 

■KESI 

92.93 

909,310 

5,440 

92.93 

909,300 

5,440 

92.93 

912,726 

5,440 

92.93 

909,184 

5,440 

92.93 

910,463 

5,440 

92.93 

5,440 

92.93 

906,425 

5,440 

92.93 

913,111 

■■SSI 

92.93 

908,408" 

5,440 

92.93 

909,967 

5,440 

92.93 

908,718 

5,440 

92.93 

908,376 

5,440 

92.92 

909,935 

5,440 

1 ,391 1  1,718,9751  3,292,9001 


1,359  1,720,580 


1,339  1,718,828 


1,326  1,718,019 


1,375  1,718,266 


1,368  1,719,412 


1,357  1,717,872 


1,359  1,719,178 


m 


1,377  1,719,547 


1,302  1,718,584 


1,375  1,718,935 


1,367  1,719,873 


1,380  1,719,466 


1,349  1,719,757 


1,361  1,718,433 


1,293  1,717,365 


1,364  1,718,581 


1,321  1,720,910 


1,365  1,720,295 


1,376 


1,717,819 


1,330  1,717,869 


1,390  1,718,936 


1,350  1,718,542 


1,367  1,718,673 


1,718,459 


1,424  1,718,922 


1,402  1,717,7781 


mm ii 


3,128,674 

3,102,043 

3,104,702 


3,223,316 


3,179,651 


3,176,295 


3,378,578 


3,323,445 


3,246,892 


3,067,133 


3,207,577 


3,258,271 


3,261,265 


3,110,609 


3,102,280 


3,051,015 


3,174,327 


3,063,736 


3,126,011 


3,166,106 


3,177,639 


3,078,273 


IBEM8M 

mm 


3,129,802 


3,233,155 


3,179,600 


1,336  1,720,902 


3,235,067 


3,364,259 


3,231,122 


3,269,965 


3,189,299 


3,132,721 


92.93 


92.93 


92.9 


92.93 


92.94 


92.93 


92.93 


92.93 


92.93 


92.93 


92.92 


92.93 


92.93 


92.93 


92.94 


92.92 


92.93 


92.93 


92.93 


92.921 


92.93 


92.93 


92.93 


92.93 


92.93 


92.93 


909,516 


912.540 


907,754 


908,808 


914,022 


908,065 


908,598 


912,191 


909,791 


907.828 


908,430 


908,417 


912,816 


911,754 


910,501 


912,738 


912,995 


912,602 


912,207 


913.369 


908,862 


909.420 


908,125 


907.483 


905,470 


913,189 


909,965 


5,4401 


5,440 


5,440 


5,440 


5,440 


5,440 


5,440 


5,440 


5,440 


5,440 


5,440 


5,440| 


5,440 


5,440 


5,440 


5,440 


5,440 


5,440 


5,440 


5,440 


5,440 


5,4401 


5,440 


1,325 


1,374 


1.399 


1,297 


1,411 


1,370 


1,376 


1,365 


1,362 


1,405 


1,378 


1,365 


1.378 


1,387 


1,358 


1,317 


1,421 


1,334 


1,347 


1,336 


1,351 


1,382 


1,362 


1.315 


1.342 


1,394 


1,383 


1 ,720,013 


1,719,753 


1.719.097 


1,718,432 


1,718,216 


1,719,103 


1,718,480 


1,717,798 


1,720,330 


1,717,959 


1,721,030 


1,720,160 


1.719.732 


1,720,4891 


1,719,102 


1 ,718,719 


1,719,267 


1,718,098 


1 ,719,068 


1 ,719,704 


1,719,208 


1,719,438 


1,719,106 


1.717.599 


1,719,344 


1,719,186 


1,718,540 


3,109,189 


3,203,944 


3.359.174 


2,939,887 


3,295,320 


3,137,401 


3,231,753 


3.202.152 


3,181,126 


3.293.213 


3,228,356 


3.197.565 


3,209,248 


3,324,458 


3.166.563 


3,020,374 


3.348.046 


3.120.212 


3,158,549 


3,035,597 


3,207,424 


3,248,156 


3,201,921 


3.059.935 


3.120.826 


3,235,963 


3.249.418 


std  dev 


92.93 

5,440 

1,361 

1,719,026 

3,186,878 

2.44E-05 

4,355,968 

0 

919 

903,875 

6,876,350,77 

0 

0 

2087 

0 

30 

951 

82924 

107 


Sample 

Time 

Sec. 

Work 

Kilo- 

Whetstones 

Detect 

Message 

s 

Sent 

Messages 

Received 

Bytes 

Sent 

Bytes 

Received 

1 

0 

967,860 

609 

2,720 

2,139,347 

816,902 

2 

0 

969,114 

609 

2,720 

2,139,052 

817,562 

3 

0 

968,895 

612 

2,720 

2,149,856 

815,847 

4 

0 

969,505 

597 

2,720 

2,099,620 

816,026 

5 

0 

967,881 

623 

2,720 

2,183,841 

815,905 

6 

0 

969,844 

623 

2,720 

2,191,876 

815,234 

7 

0 

967,739 

643 

2,720 

2,255,008 

816,047 

8 

0 

969,035 

632 

2,720 

2,213,479 

815,780 

9 

0 

967,331 

609 

2,720 

2,133,246 

815,642 

10 

0 

966,925 

643 

2,720 

2,260,495 

816,099 

11 

0 

968,147 

587 

2,720 

2,054,101 

816,343 

12 

0 

968,916 

605 

2,720 

2,125,097 

816,584 

13 

0 

968,077 

641 

2,720 

2,256,241 

815,366 

14 

0 

969,994 

607 

2,720 

2,131,274 

815,849 

15 

0 

969,115 

614 

2,720 

2,152,732 

814,559 

16 

0 

968,243 

598 

2,720 

2,093,913 

815,493 

17 

0 

968,357 

648 

2,720 

2,276,068 

815,477 

18 

0 

969,747 

595 

2,720 

2,089,478 

816,510 

19 

0 

967,663 

616 

2,720 

2,156,847 

816,450 

20 

0 

968,647 

575 

2,720 

2,016,793 

815,718 

21 

6 

968,059 

642 

2,720 

2,253,705 

815,961 

22 

0 

968,520 

631 

2,720 

2,213,249 

815,290 

23 

0 

969,410 

573 

2,720 

2,013,320 

817,569 

24 

o 

969,958 

634 

2,720 

2,231,761 

815,298 

25 

0 

968,302 

609 

2,720 

2,139,430 

815,688 

26 

0 

969,657 

617 

2,720 

2,167,783 

815,370 

27 

6 

967,534 

615 

2,720 

2,162,390 

814,377 

28 

0 

967,103 

624 

2,720 

2,195,646 

817,230 

29 

0 

969,325 

589 

2,720 

2,066,123 

814,812 

30 

0 

968,224 

604 

2,720 

2,123,249 

815,364 

31 

0 

969,459 

605 

2,720 

2,125,289 

814,233 

32 

0 

967,721 

624 

2,720 

2,188,345 

816,365 

33 

0 

968,211 

622 

2,720 

2,184,035 

815,886 

34 

0 

967,027 

653 

2,720 

2,296,915 

816,192 

35 

0 

968,491 

648 

2,720 

2,269,414 

815,997 

36 

0 

967,987 

625 

2,720 

2,196,295 

816,609 

108 


968,2141 


967,201 


967,267 


969,012 


969,007 


967,6601 


969,781 


969,165 


968,406 


966,7061 


966,927 


967,287 


968,882 


967,958 


967,493 


967,601 


967,418 


968,882 


968,476 


966,6761 


968,3761 


969,271 

968,765' 

970,225' 

969,714' 

968,060' 

969,181' 

967,750' 

968,728' 

968,138' 


2,720 


2,720 


2,720 


2,720 


2,720 


2,720 


2,720 


2,720 


2,720 


2,720 


2,720 


2,720 


2,720 


2,720 


2,118,720 


2,075,610 


2,067,811 


2,103,921 


2,187,772 


2,014,336 


2,162,882 


814,783 


816,483 


816,074 


815,514 


816,581 


815,653 


815,397 


2,280,983 


2,301,536 


2,224,440 


2,100,463 


2,156,007 


wteiaaiwaBMiial 


816,116 


816,243 


815,991 


815,785 


816,459 


816,659 


2,7201  2,119,660  816,119 


815,447 


2,140,108  815,433 


2,1 16,654  814,720 


2,244,263  815,726 


2,720 


2,720 


2,720 


2,720 


2,720 


2,720 


2,133,171  816,601 


814,497 


2,103,552  815,697 


2,250,124 


2,720 


2,720 


2,720 


2,720 


2,720 
2,720' 
2,720  ~ 
2,720  ~ 
2,720' 
2,720 ' 
2,720 ' 
2,720' 
2,720 ' 


2,093,737 


2,251,128 


2,135,198 


2,208,450 

2,226,884' 

2,313,910' 

2,140,301' 

2,263,110' 

2,169,015' 

2,125,509' 

2,116,663 

2,161,699 


815,932 


815,163 


816,422 


815,525 

816,845 

816,209 

815,149 

817,246 

816,237 

816,657 

816,198 

816,017 


109 


79 

80 
81 
82 

83 

84 

85 

86 

87 

88 

89 

90 

91 

92 

93 

94 

95 

96 

97 

98 

99 

100 
101 
102 
103 


_o 

_0 

_0 

a 

_o 

_o 

jd 

o 

0 

0 

0 

0 

0 

0 

0 

0 

0 

0 

0 

0 

0 

0 

0 

0 


969,077 

968,052 

969,984 

967,115 

968,240 

968,419 

966,553 

967,977 

968,077 

968,530 

967,768 

968,992 

968,974 

969,113 

968,536 

967,947 

969,312 

967,826 

969,540 

968,238 

969,687 

966,898 

968,157 

968,523 

968,855 


Average 

0.00 

968,361 

Variance 

0 

735,880 

std  dev 


0 


858 


673 

2,720 

2,362,863 

815,828 

541 

■IK SH3 

1,898,144 

815,388 

635 

2,720 

2,227,705 

814,544 

586 

2,720 

2,059,820 

816,150 

626 

2,720 

2,199,494 

815,942 

621 

2,720 

2,178,676 

815,743 

611 

2,720 

2,147,854 

EEEH 

636 

2,720 

2,237,061 

815,046 

625 

2,720 

2,194,880 

816,778 

618 

2,720 

2,171,056 

816,299 

616 

2,720 

2,161,025 

816,870 

665 

2.720 

2,332,101 

817,174 

605 

2,720 

2,129,122 

815,901 

569 

2,720 

1,991,398 

816,705 

655 

2,720 

2,296,613 

815,526 

602 

2,720 

2,111,946 

815,701 

610 

2,720 

2,143,728 

wmtm 

561 

2,720 

1,968,391 

816,898 

634 

2,720 

2,221,693 

816,378 

630 

2,720 

2,212,557 

816,721 

623 

2,720 

2,186,857 

816,800 

587 

2,720 

2,058,513 

814,473 

597 

2,720 

2,097,492 

814,929 

619 

2,720 

2,169,853 

815,223 

628 

2,720 

2,209,622 

815,884 

616 

2,720 

2,162,317 

815,936 

525 

0 

6,563,514,7 

76 

491,281 

23 

0 

81016 

701 

no 


FP 


Sample 

Time 

Work 

Message 

e 

Message 

c 

Bytes 

Bytes 

Sec. 

Kilo-Whetstones 

o 

Sent 

o 

Received 

Sent 

Received 

1 

0 

887,515 

754 

2,720 

1,059,819 

815,923 

2 

0 

887,030 

722 

2,720 

1,013,338 

815,687 

3 

0 

887,382 

727 

2,720 

1,021,757 

815,556 

4 

0 

887,248 

717" 

2,720 

1,007,414 

814,844 

5 

0 

887,703 

755 

2,720 

1,059,457 

815,824 

6 

0 

886,247 

781 

2,720 

1,097,719 

816,108 

7 

0 

885,869 

742 

2,720 

1,041,685 

816,107 

8 

0 

884,721 

747 

2,720 

1,048,839 

816,271 

9 

0 

887,758 

756 

2,720 

1,063,087 

816,340 

10 

0 

887,602 

737 

2,720 

1,035,221 

816,787 

11 

0 

887,343 

801 

2,720 

1,123,590 

814,884 

12 

0 

887,118 

750 

2,720 

1,050,572 

816,690 

13 

6 

886,285 

756 

2,720 

1,066,135 

815,911 

14 

0 

884,872 

768 

2,720 

1,075,743 

815,771 

15 

0 

887,056 

763 

2,720 

1,072,398 

816,581 

16 

0 

885,443 

743 

2,720 

1,043,474 

816,485 

17 

0 

887,015 

737 

2,720 

1,033,547 

815,436 

18 

0 

886,443 

703 

2,720 

986,008 

816,206 

19 

0 

887,982 

761 

2,720 

1,067,512 

816,420 

20 

0 

887,737 

738 

2,720 

1,037,326 

816,800 

21 

0 

886,123 

707 

2,720 

993,759 

816,861 

22 

0 

886,038 

744 

2,720 

1,045,106 

815,940 

23 

0 

884,761 

744 

2,720 

1,044,964 

816,743 

24 

0 

886,031 

736 

2,720 

1,032,146 

815,833 

25 

0 

886,092 

748 

2,720 

1,051,341 

816,906 

26 

0 

884,893 

742 

2,720 

1,041,047 

816,595 

27 

0 

886,373 

745 

2,720 

1,048,631 

816,612 

28 

0 

887,582 

723 

2,720 

1,011,942 

814,505 

29 

0 

887,656 

759 

2,720 

1,066,250 

814,981 

30 

0 

888,279 

752 

2,720 

1,057,758 

816,145 

31 

0 

886,932 

701 

2,720 

986,533 

815,083 

32 

0 

886,700 

729 

2,720 

1,024,408 

815,587 

33 

0 

887,984 

754 

2,720 

1,059,773 

816,033 

34 

0 

887,166 

751 

2,720 

1,054,583 

816,507 

35 

0 

885,652 

743 

2,720 

1,045,742 

815,938 

36 

0 

887,282 

716 

2,720 

1,006,242 

815,643 

in 


°  °  °  O  0|0|0|0|0|0|0|0|0|0|0|0|0|0|0|0|0|0|0|0|0|0  o  o  o  o  o  o  o  o  o  o  o  o  o  o  o  o 


886,591 


887,254 


887,196 


888,558 


887,130 


887,402 


885,892 


887,819 


886,387 


886,709 


886,179 


886,188 


885,991 


886,161 


886,205 


886,459 


886,052 


887,444 


885,646 


887,722 


886,186 


887,946 


887,783 


884,837 


887,288 


886,361 


2,7201 


2,720 


2,720 


2,720 


2,7201 


2,7201 


2,720 


2,720 


2,720 


2,720 


2,720 


2,7201 


976,921 


1,074,808 


1,055,656 


1,021,997 


1,057,544 


1,116,462 


1,038,481 


1,044,872 


1,120,747 


1,044,293 


1,044,484 


987,502 


1,073,634 


1,033,285 


1,085,363 


1,053,825 


989,778 


1,088,388 


1,072,246 


1,049,616 


1,055,995 


967,770 


1,017,235 


994,297 


815,504 


817,057 


815,714 


815,465 


816,719 


815,435 


815,988 


817,024 


815,326 


816,516 


815,759 


816,435 


816,307 


816,089 


816,174 


816,469 


816,948 


816,946 


815,960 


815,605 


1,066,308 


1,057,7041 


965,977 


1,055,365 


1,057,665 


1,003,691 


816,654 


816,062 


815,082 


814,882 


816,213 


815,570 


2,7201 


2,720 


1,053,129 


1,030,055 


1,073,133 


1,113,253 


1,028,887 


1,042,012 


1,028,588 


1,013,726 


1,064,229 


816,573 


814,574 


815,673 


815,589 


817,460 


816,775 


79 

0 

885,433 

726 

2,720 

1,018,695 

816,229 

80 

0 

HESSs! 

1,062,495 

816,004 

81 

0 

885,973 

WKKL 5E3 

1,090,191 

816,632 

82 

0 

886,893 

784 

2,720 

1,099,501 

815,913 

83 

0 

886,270 

750 

2,720 

1,054,275 

84 

0 

886,625 

744 

2,720 

1,045,316 

815,015 

85 

0 

886,478 

751 

2,720 

1,055,064 

817,447 

86 

WEE& 1 

1,078,632 

87 

0 

887,222 

753 

2,720 

1,055,524 

817,212 

88 

0 

885,723 

747 

2,720 

1,048,349 

816,821 

89 

0 

762 

2,720 

1,070,271 

815,822 

90 

886,279 

722 

2,720 

HuilsSUsi 

91 

0 

886,047 

753 

2,720 

1,059,169 

816,161 

92 

0 

748 

2,720 

1,050,048 

814,974 

93 

0 

886,694 

■KsZXED 

94 

0 

884,943 

732 

1,029,6101 

815,357 

95 

0 

737 

2,720 

1,036,373 

815,805 

96 

0 

888,233 

775 

2,720 

■sEf m 

97 

0 

887,956 

717 

1,007,347 

815,790 

98 

0 

885,646 

752 

2,720 

1,057,711 

815,677 

99 

0 

885,840 

739 

2,720 

1,036,856 

815,266 

100 

0 

887,529 

728 

WMEEsl 

1,022,462 

816,086 

101 

0 

886,783 

745 

2,720 

1,044,806 

817,375 

102 

0 

886,614 

775 

WESiL 3 

1,088,414 

816,923 

103 

0 

888,059 

755 

2,720 

1,061,924 

815,616 

Average 

Variance 

std  dev 


0.00 

■MMB&l'iYiW:! 

745 

2,720 

1,046,334 

816,050 

0 

518 

0 

1,018,850,6 

00 

0 

899 

23 

0 

31919 

675 
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